Image processing apparatus, an image processing method, a medium on which an image processing control program is recorded, an image evaluation device, an image evaluation method and a medium on which an image evaluation program is recorded

ABSTRACT

A method for image processing image data having a plurality of picture elements. While scanning the image data, a scanned picture element is determined to be sampled or not according to a plurality of sampling methods until the scanned picture element is displaced to a next picture element (to be scanned) of the image data.

This is a continuation of Ser. No. 13/081,027 filed Apr. 6, 2011, whichis a continuation of Ser. No. 11/491,278 filed Jul. 24, 2006, which is adivisional of application Ser. No. 10/742,718 filed Dec. 22, 2003, nowU.S. Pat. No. 7,259,894 issued Aug. 21, 2007, which is a divisional ofapplication Ser. No. 09/093,094 filed Jun. 8, 1998, now U.S. Pat. No.6,738,527, issued May 18, 2004, the disclosures in which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an image processing method wherein an optimumimage processing is performed automatically on photograph image datasuch as digital photograph image, and the image is evaluated, to animage processing apparatus, a medium on which an image processingcontrol program is recorded, an image evaluation device, an imageevaluation method, and a medium on which an image evaluation program isrecorded.

2. Description of the Prior Art

Various kinds of image processing may be performed on digital imagedata, i.e., in which processing: contrast may be increased; color may becorrected; or lightness may be corrected. This image processing canusually be performed with a microcomputer. An operator confirms theimage on a monitor, the necessary image processing is selected, andimage processing parameters are determined.

In recent years various types of image processing techniques have beenproposed, and are now having considerable impact. However, a humanoperator is still required when it is a question of which technique toapply, and to what extent it should be used. This is because it wasotherwise impossible to determine which digital image data had to besubjected to image processing. For example, in the case of imageprocessing to correct the lightness of an image, the screen is madelighter if it is dark on the whole, and is made darker if it is toolight.

Now, consider the case of a photographic image of a person filmed atnight, where the background is near to pitch-darkness but the person inthe picture has been well photographed. If this photograph isautomatically corrected, it is attempted to make the image brighter dueto the fact that the background is pitch black, so the final imageappears as if the photo was taken in the daytime.

In this case, if a human operator is involved, he pays attention only tothe person in the picture. If the image of the person is dark, it wouldbe made a little brighter, conversely darkening would be selected if theeffect of flash, etc., was too bright.

Hence, there was a problem in the prior art in that a human operator hadto participate to determine the important part (referred to hereafter asthe “object”) of a photographic image.

However, even when the importance of the image is evaluated by sometechnique, the determination process is performed in picture elementunits, and varying the importance in real time causes an increase incomputation.

SUMMARY OF THE INVENTION

It is therefore an object of this invention, which was conceived in viewof the aforesaid problems, to provide an image processing method whichpermits an important part of a photographic image such as a digitalphotograph image to be detected, and an optimum image processing to beautomatically selected, to provide an image processing apparatus, and toprovide a medium on which an image processing control program isrecorded.

In order to achieve the aforesaid object, this invention is an imageprocessing apparatus into which photographic image data comprising dotmatrix picture elements is input, and which performs predetermined imageprocessing on the input data. The apparatus comprises an imageprocessing acquiring unit which acquires the aforesaid photographicimage data, an image processing indicator specifying unit which performspredetermined summation processing on picture elements and specifies animage processing indicator based on the acquired image data, and aprocessing unit which determines image processing contents based on thespecified image processing indicator, wherein the aforesaid imageprocessing indicator specifying unit comprises an object determiningunit which determines picture elements having a large image variationamount to be those of the object, and the aforesaid processing unitdetermines image processing contents based on image data for pictureelements determined to be those of the object and performs imageprocessing on the determined contents.

Herein, it is assumed that in the case of an image of a photographicimage of a person, the person is usually photographed in the center ofthe field. Therefore, the person is brought into focus to give a sharpimage. When the image is sharp, the outline part becomes clear, and theamount of image variation becomes large. As a result, there is anextremely high possibility that there will be no error if it is assumedthat picture elements with a large image variation amount are those ofthe original object which has been brought into focus.

In the invention thus comprised, in the image processing indicatorspecifying unit based on photographic image data comprising dot matrixpicture elements acquired in the image data acquiring unit,predetermined summation processing is performed on picture elements.This summation processing may take various forms, and the imageprocessing indicator is basically specified based on the summationresult. In the processing unit, the image processing contents aredetermined based on the specified indicator, and the determined imageprocessing is performed.

This means that useful information about the corresponding image can beobtained by performing summation processing on dot matrix photographicimage data, and image processing is performed on the data. In this way,the image processing indicator is specified by actual photographic imagedata, so optimum image processing can be performed even without humanintervention.

The photographic image data comprises dot matrix picture elements andimage processing is performed in picture element units, but first, inthe object determining unit, picture elements having a large imagevariation are determined to be those of the object. Image processingcontents are then determined in the processing unit based on image datafor picture elements determined to be those of the object, and imageprocessing is performed based on the determined contents.

Therefore, according to this invention, the determination of the object,which in the past required human participation, can be automated bydetermining the object as picture elements with a large image variation.The invention therefore provides an image processing method wherebyoptimum image processing can be performed by suitably modifying theimage processing contents according to the object.

This method for specifying an image processing indicator from actualphotographic data may of course be applied not only to a real device butalso to a system on the method. In such a sense, this invention is alsoan image processing method wherein photographic image data comprisingdot matrix picture elements is input, and predetermined image processingis performed, this image processing method comprising an image dataacquiring step for acquiring the aforesaid photographic image data, animage processing indicator specifying step for performing apredetermined summation processing on picture elements based on thisacquired image data, and specifying an image processing indicator, and aprocessing step for determining image processing contents based on thespecified indicator, and performing image processing, wherein theaforesaid image processing indicator specifying step comprises an objectdetermining step wherein picture elements having a large image variationamount are determined to be those of the object, and wherein in theaforesaid processing step, image processing contents are determinedbased on image data for picture elements determined to be those of theobject, and image processing is performed based on the determined imageprocessing contents. In this case, the apparatus offers all theattendant benefits and advantages of the invention.

This apparatus for determining an object and performing image processingmethod may be implemented by a stand-alone apparatus as mentioned above,or may be incorporated in another instrument which comprises such anapparatus. In other words, the scope of this invention covers variousforms of implementation. It may also be implemented by hardware orsoftware, and can be modified as necessary.

When the apparatus for implementing the concept of this invention isimplemented by software, the invention applies equally to media on whichthis software is recorded and which can be used in exactly the same way.In this sense, this invention is also a recording medium whereon animage processing control program is recorded for inputting photographicimage data comprising dot matrix picture elements by a computer, andwhich performs predetermined image processing on the input data. Thecontrol program comprises an image processing indicator specifying stepfor acquiring the aforesaid photographic image data, an image processingindicator specifying step for performing predetermined summationprocessing on picture elements and specifying an image processingindicator, and a processing step for determining image processingcontents based on the specified image processing indicator, wherein theaforesaid image processing indicator specifying step comprises an objectdetermining step which determines picture elements having a large imagevariation amount to be those of the object, and in the aforesaidprocessing step, image processing contents are determined based on imagedata for picture elements determined to be those of the object, andimage processing is performed on the determined contents. In this case,the recording medium offers all the attendant benefits and advantages ofthe invention.

The recording medium may of course be a magnetic recording medium, anoptomagnetic recording medium, or any kind of recording medium which maybe developed in the future. It will of course be understood that themedium may be a first copy or second copy, and that a telecommunicationline may also be used to supply the program. In this case, there is nodifference regarding the application of the invention. There is also nodifference if the program is written on a semiconductor chip.

There is no difference as regards the concept of the invention even ifone part is software, and one part is implemented with hardware, or whenit is in such a form that one part is stored on a recording medium whichcan be read when necessary.

Photographic image data means image data obtained when it is attemptedto take a photograph of a real object. Image processing tries to correctimages by comparing the images with the real objects from which theywere acquired. The invention therefore applies not only to naturalobjects but also to manmade ones. More specifically, this includes imagedata read by a scanner, or image data captured by a digital camera.

Various techniques may be employed to determine the variation of pictureelements in the object determining step. A further object of thisinvention is to give a specific example of this.

In the image processing method provided by this invention, in theaforesaid object determining step, the amount of variation of pictureelements is determined based on a difference between adjacent pictureelements.

Hence according to this invention, in the object determining unit wherean image variation amount is determined, the determination is performedbased on a difference of image data between adjacent picture elements.When there is a fixed interval between picture elements as in the caseof a dot matrix, the difference of data between adjacent pictureelements is directly proportional to a first order differential. Thisdifference can be taken as the variation amount of the image. In thiscase the difference is regarded as the magnitude of a vector, and thevector may also be constructed by considering adjacent directions.

According to this invention, only the difference of image data betweenadjacent picture elements is found. Computation is therefore easy, andthe processing amount for object determination can be reduced.

The determination of an object is of course not limited to thistechnique, and it is a further object of this invention to provide otherexamples.

According to the image processing apparatus provided by this invention,in the aforesaid object determining unit, the criterion for determiningwhether or not there is a large image variation amount changes accordingto the position of the image.

In the case of a photograph for example, a person is often photographedin the center. In this case it may be said that in order to determineimage processing content, the picture elements to be determined as theobject should be selected from the central part of the field. However,it may be said that whether or not there is a large variation in theimage depends on a difference from a comparison value, and there is noreason why such a value always has to be constant.

Therefore, to determine whether or not there is a large image amountaccording to this invention, in the object determining unit, thiscriterion is altered depending on the position of the image, thecriterion for each position being compared with the image variationamount of each picture element.

Hence, according to this invention, the assessment of image variationchanges depending on the position of the image, and a highly flexibledetermination which considers image composition is thus possible.

The criterion can be altered in various ways. As one example, a certaintrend may be ascertained, or alternatively, a trend which causes achange may be read from the image.

A further object of this invention is to provide an example of theformer.

In the image processing apparatus provided by this invention, in theobject determining unit, the aforesaid criterion is set lower for thecentral part than for the edges of the image.

By setting the criterion lower for the center than for the edges, it iseasier to determine the center part of the image as the object even ifthe variation amount at the center and at the edges is approximately thesame. Therefore if there is an image of a person in the central part,the picture elements of this person will be determined as the objectmore frequently.

According to this invention, a determination can be made which givesmore weight to the center area of a photograph, and a large amount ofimage data can be effectively processed.

A further object of this invention is to provide an example of thelatter when the criterion is varied.

According to the image processing apparatus, provided by this invention,in the aforesaid object determining unit, the above criterion is basedon the distribution of the aforesaid image variation amount at differentpoints on the image.

Hence according to this invention, in the object determining unit, thedistribution of image variation is found in each part of the image, andthe aforesaid criterion is determined after finding this distribution.Subsequently, a comparison is made with this criterion to determinewhether or not the picture elements are those of the object.

According to this invention, as the object is determined taking accountof the distribution of image variation for picture elements, the imagedata can be treated flexibly.

When the criterion is determined based on distribution, it may beconsidered that there is a high possibility of finding the object in apart where there are many picture elements with a large variationamount, and the criterion maybe set low.

Alternatively, basic setting patterns may first be prepared according toa variation distribution pattern, and a basic setting pattern may thenbe chosen based on the detected distribution pattern.

At the same time, assuming that the image processing indicatorspecifying unit comprises such an object determining unit, the imageprocessing contents may be determined based on image data which isdetermined to be that of the object, and image processing may then beperformed on the determined contents, there being no limitation on thespecific processing method employed. For example, a luminancedistribution of picture elements determined to be those of the object isfound, and if the luminance distribution range is enlarged in apredetermined proportion when the luminance distribution is narrow,image processing to increase contrast is performed. If the luminancedistribution of the object seems dark on the whole, a correction may bemade to make it lighter. The color distribution of picture elementsdetermined to be those of the object is found, and it is determinedwhether or not the grey balance is off. If it seems to be off, tonecurves are used to modify the grey balance.

Hence, the importance of the image has an effect even if the image datais summed in order to specify the image processing indicator. However,even if the importance of the image is determined by some technique, thework is carried out in picture element units, so varying the importanceof an image in real time implies an increase of computational amount.

A further object of this invention is to consider the importance ofphotographic image data such as digital photograph images in relativelysimple terms, and perform optimum image processing automatically.

The image processing apparatus provided by this invention is anapparatus for inputting photographic image data comprising dot matrixpicture elements, and performing predetermined image processing. Thisimage processing apparatus comprises an image data acquiring unit foracquiring the aforesaid photographic image data, an image processingindicator specifying unit which performs a predetermined summationprocessing on picture elements based on this acquired image data andspecifies an image processing indicator, and a processing unit whichdetermines image processing contents based on the specified indicatorand performs image processing. The aforesaid image processing indicatorspecifying unit comprises a feature amount uniform sampling unit whichdetermines an image processing intensity by uniformly sampling a featureamount over a whole screen, and a feature amount weighting reevaluationunit which reevaluates the feature amount sampled in the feature amountsampling unit with a predetermined weighting. In the aforesaid imageprocessing unit, the image processing intensity is determined based onthe reevaluated feature amount, and image processing is performed withthe determined intensity.

According to the invention having the above construction, photographicimage data comprises dot matrix picture elements, and in the featureamount uniform sampling unit, the feature amounts of picture elementsare uniformly sampled over the whole screen. In the feature amountweighting reevaluation unit, the feature amounts that are sampled inthis feature amount uniform sampling unit are reevaluated with apredetermined weighting. Then, in the processing unit, the imageprocessing intensity is determined based on the feature amounts thathave been reevaluated in this way, and image processing is performed.

In other words, as the sampling is uniform over the whole screen and apredetermined weighting is applied after sampling, the feature amountsobtained as a result are different from what is obtained by uniformsampling over the whole screen.

According to this invention the sampling in the sampling stage isuniform over the whole screen, so the computational amount is not toohigh. At the same time, by applying a predetermined weighting aftersampling, irrelevant evaluation is not made as it would be if thepicture elements were merely sampled uniformly without weighting. Theinvention therefore provides an image processing apparatus in whichoptimum image processing can be performed automatically.

It will be understood that the technique of performing a uniformsampling in the sampling stage and applying a predetermined weightingthereafter, may be applied not only to a real device but also to asystem in both of which cases it has all the attendant benefits andadvantages of this invention. As a specific example of the concept ofthis invention, when the image processing apparatus is implemented interms of software, there naturally exist recording media on which thesoftware is recorded which can be used to perform the function of theinvention.

The feature amount uniform sampling unit uniformly samples featureamounts over the whole screen, for determining the image processingintensity. For this purpose, all picture elements over the whole screencan be sampled, but it is not necessary to sample all of the pictureelements if the sampling is uniform.

A further object of this invention is to provide an example of thelatter case.

According to the image processing apparatus of this invention, in theaforesaid feature amount uniform sampling unit, the aforesaid featureamounts are sampled for selected picture elements after uniformlythinning out the picture elements according to predetermined criteria.

According to this invention, by thinning out the picture elementsaccording to predetermined criteria, the number of picture elements tobe processed is reduced, and the aforesaid feature amounts are sampledfrom the remaining elements.

Herein, the term “uniform thinning” comprises the case where pictureelements are selected at a fixed interval, and the case where they areselected at random.

According to this invention, as the picture elements are thinned outwhen the feature amounts are uniformly sampled, the processing amount isreduced.

The sampled feature amounts are reevaluated by a predetermined weightingin the feature amount weighting reevaluation unit. The sampled featureamounts are in picture element units, but the weighting can be appliedeither to picture element units or to suitable aggregates of pictureelements.

A further object of this invention is to provide an example of thelatter case.

According to the image processing apparatus of this invention, in theaforesaid feature amount uniform sampling unit, feature amounts aresampled in area units that are divided according to predeterminedcriteria, and in the aforesaid feature amount weighting reevaluationunit, a weighting is set for each area and the feature amounts are thenreevaluated.

The invention as formulated hereabove assumes weightings in area unitsof the image that are divided according to predetermined criteria. Inthe feature amount uniform sampling unit, feature amounts are sampled inthese area units, while in the aforesaid feature amount weightingreevaluation unit, the feature amounts are reevaluated with weightingsset for each area.

The division of these areas may always be constant, or it may be made tovary for each image. In this latter case the division method may bechanged according to the contents of the image.

According to this invention, as the weighting is made to vary for eacharea, the computation is relatively simple.

Any type of weighting technique can be employed provided thatreevaluation is performed without merely performing uniform sampling.

A further object of this invention is to provide an example of this.

According to the image processing apparatus of this invention, in theaforesaid feature amount weighting reevaluation unit, the aforesaidweighting is made to vary by a correspondence relation determined by theposition of picture elements in the image.

In the case of a photograph, the person is usually in the center.Therefore, by weighting the central part of the image more heavily afteruniformly sampling feature amounts from the whole image, the featureamounts sampled from picture elements relating to the person areevaluated to be larger.

When for example according to the invention thus comprised, theweighting of the central part of the image is heavier and the weightingof the surroundings is lighter, in the feature amount weightingreevaluation unit, the position of picture elements in the image isdetermined, and a reevaluation is made using a weighting which variesaccording to this position.

Hence according to this invention, as the weighting is determinedaccording to the position of picture elements, the computation isrelatively simple.

The weighting technique is of course not limited to this method, and afurther object of this invention is to provide other examples.

According to the image processing apparatus provided by this invention,in the aforesaid feature weighting reevaluation unit, the imagevariation amount is found, and a heavier weighting is given to partswhere the image variation amount is larger.

In the invention thus comprised, the image variation amount is foundbefore performing reevaluation in the feature amount weightingreevaluation unit. The image variation amount is also known as imagesharpness, and as the outline is sharper the better the focus, thevariation is large where the image is in focus. On a photograph, thepart which is in focus is the subject, and the part which is not infocus is considered to be the background. Therefore, places where thereis a large image variation are considered to correspond to the subject.In the feature amount weighting reevaluation unit, the same result assampling a large number of feature amounts is obtained by applying heavyweighting to parts where there is a large image variation.

According to this invention, as the weighting is varied depending onimage sharpness, different targets can be precisely identified andfeature amounts can be sampled for different images.

As another example of a weighting technique, in the weightingreevaluation unit of the image processing apparatus of the invention,the chromaticity of picture elements is found, a number of pictureelements is found for which the chromaticity lies within thechromaticity range of the target for which it is desired to sample afeature amount, and heavier weighting is applied to parts where thereare many of these picture elements.

Hence according to this invention, in the feature amount weightingreevaluation unit, the chromaticity of picture elements is found. Inimage processing, an object can sometimes be specified by a specificchromaticity. For example, there is no reason why a person could not beidentified by looking for skin color, but it is difficult to specifyskin color as color data also contain luminance elements. Chromaticityon the other hand represents an absolute proportion of a colorstimulation value, and it is not controlled by luminance. Therefore animage of a person could be determined if the chromaticity was within aspecified range that can be taken as indicative of skin color. Thisreasoning may of course also be applied to the green of the trees or theblue of the sky.

As specified objects can be sorted by chromaticity according to thisinvention, different targets may be precisely sampled depending on theirimages and feature amounts sampled.

Therefore in a feature amount weighting reevaluation unit, when thechromaticity found for picture elements is within a chromaticity rangefor a target from which it is intended to sample feature amounts, pluralpicture elements are counted. When the number of picture elements islarge, this part of the image is determined to be the target, heavyweighting is applied, and a large feature amount is sampled from thetarget.

This weighting technique is not necessarily the only alternative, and itis a further object of this invention to provide a suitable example ofoverlapping methods.

According to the image processing apparatus of this invention, in theaforesaid feature amount weighting reevaluation unit, temporaryweightings are applied based on a plurality of factors, and thesefactors are then added according to their degree of importance to givefinal weighting coefficients.

According to the invention as thus comprised, in the feature amountweighting reevaluation unit, temporary weighting coefficients are foundseparately based on a plurality of factors, and the weightings are addedaccording to their degree of importance so as to reevaluate the sampledfeature amounts as final weighting coefficients. Therefore, it may occurthat even when a large weighting is assigned by one weighting method inthe evaluation stage, if the method does not have a large importance,the final weighting which is assigned is not large. Moreover, it mayoccur that even if there is a large difference between weightingmethods, image parts which are evaluated to have an average or higherweighting also have a large final weighting.

According to this invention, plural weighting techniques are suitablycombined so that a suitable feature amount evaluation can be performed.

Therefore if the image processing indicator specifying unit itselfcomprises plural forms, it is not necessary to perform image processingwith only one of these forms.

However even if there are some cases where it is desirable to performimage processing using the feature amount of the object, there are othercases where it is desirable to perform image processing using an averagefeature amount for the whole photographic image. For example, when aphotograph is taken of a person, the person may not always be thelightest (highlighted) part of the picture. Therefore if attention ispaid only to the person in the picture and contrast is increased, thehighlighted part of the background will be too white. In this case, abetter result would be obtained by paying attention to the wholephotographic image.

Hence when image processing is performed, it is still necessary toselect an optimum feature amount.

It is a further object of this invention to automatically select anoptimum feature amount according to image processing technique.

In the image processing apparatus according to this invention,photographic image data comprising dot matrix picture elements is input,and predetermined image processing is performed. This image processingapparatus comprises an image data acquiring unit for acquiring theaforesaid photographic image data, an image processing indicatorspecifying unit for performing a predetermined summation processing onpicture elements based on this acquired image data, and specifying animage processing indicator, and a processing unit for determining imageprocessing contents based on the specified indicator, and performingimage processing. The aforesaid image processing indicator specifyingunit comprises an evaluation unit for obtaining a feature amount byinputting the aforesaid photographic data, summing the image data forall picture elements, and obtaining a feature amount according to pluralpredetermined evaluation criteria. In the aforesaid processing unit, theimage data can be converted by plural techniques, and the featureamounts obtained in the aforesaid evaluation unit used according to theparticular technique.

According to this invention, image data from a photographic imagecomprising dot matrix picture elements is input in this manner, and afeature amount is obtained according to plural evaluation criteria bysumming image data for picture elements in the evaluation unit. In theprocessing unit, in converting the image data by plural techniques, thefeature amounts obtained in the evaluation unit according to eachtechnique are then used to convert the data depending on the technique.

Specifically, although there are cases where the image data is bestconverted using a feature amount centered on the object such as inlight/dark correction, there are other cases where image data is betterconverted using a feature amount centered on the whole image such aswhen contrast is increased, and the image data conversion may beperformed by suitably selecting these plural feature amounts.

According to this invention, when feature amounts are obtained accordingto plural evaluation criteria and image processing is performed byplural methods, the feature amounts used depend on the method, so imageprocessing may be performed based on an optimum evaluation criterion.

When image data is converted, the feature amounts should be such thatthey can be used to identify the features of the image, and there is noneed to specify the type of image. For example this also includesindicators such as luminance histograms which identify whether the imageis to be considered as light or dark, there being no need to obtain theidentification result that the image is light or dark. Apart fromlightness, the indicator may of course specify whether or not the imageis sharp, or it may be an indicator to identify vividness.

There is also no particular limitation on the way in which theevaluation unit and processing unit are applied. For example, assumingthat plural image processing methods are used, plural feature amountsmay be obtained and stored according to plural evaluation criteria, andthe image data converted by suitably selecting the feature amount in theprocessing unit as necessary. As another example, image data for pictureelements may be summed by a predetermined criterion in the evaluationunit so as to obtain a suitable feature amount on each occasion thatimage processing is performed in the aforesaid processing unit.

It will of course be understood that these plural image processingmethods based on feature amounts obtained by plural different evaluationcriteria, may also be applied not only to an actual device but also to asystem both of which are then a valid form of the invention. When theimage processing methods are implemented by software as specificexamples of the concept of the invention, there naturally exist media onwhich the software is recorded which then offer all the attendantadvantages thereof.

The evaluation criterion used to obtain feature amounts in the aforesaidevaluation unit will depend on the image processing that is to beperformed, and while there are some cases where it is desirable toconcentrate on the object for image processing, there are some caseswhere it is not as described above.

It is a further object of this invention to provide an example of theformer case.

In the image processing apparatus according to this invention, theaforesaid evaluation unit comprises an evaluation unit wherein an objectin a photographic image is sampled, and image data for picture elementsof this object is summed to obtain a feature amount, and in theaforesaid processing unit, in one processing method, the feature amountobtained from object picture elements is used when the feature amountfor the central part of the image data is used.

According to the invention thus comprised, when image data is convertedbased on the feature amount of the central part of the image data in theaforesaid processing unit, the object in the photographic image issampled in the aforesaid evaluation unit, and the feature amount isobtained by summing image data for object picture elements according topredetermined criteria.

Herein, the central part of the image data has the following meaning.For example, when it is determined whether a given photograph is lightor dark, it is easily appreciated that the determination canconveniently be based on the intermediate density of the image. Thisintermediate density may also be referred to as a median in a luminancedistribution, i.e. the center of the luminance distribution, and in thissense it is referred to as the central part of the image data. Then, ifthere is an object in the photographic image, it may be said that thereis a definite necessity to perform light/dark correction in line withthe lightness of this object.

This invention is suitable for the case when image processing isperformed based on the feature amount of the central part of the imagedata.

Any of the aforesaid techniques may be applied as the basic techniquefor sampling the object. As an example, in the aforesaid evaluationunit, picture elements for which there is a large variation of imagedata between adjacent picture elements are sampled as the object. Whenpicture elements are aligned at a fixed interval apart as in the case ofa dot matrix image, the difference of image data between adjacentpicture elements is proportional to a first order differential. Thisdifference may be determined as the image variation amount. In thiscase, the difference may be regarded as the magnitude of a vector, andthe vectors constructed taking account of adjacent directions. If thisis done it is sufficient to determine the difference of image data foradjacent picture elements, computing is easy, and the processing fordetermining the object is reduced.

As another example, in the aforesaid evaluation unit, picture elementsfor which the chromaticity is within a predetermined range may besampled as the object. In this case, in the aforesaid evaluation unit,the chromaticity of picture elements is found. The chromaticityrepresents an absolute proportion of a color stimulation value, and itis not affected by lightness. Therefore the object in the image can beseparated by possible range of chromaticity. For example, there is thechromaticity range for skin color, or the chromaticity range for thegreen of the trees. As this can be said for chromaticity, in theaforesaid evaluation unit, picture elements for which the chromaticitylies within a predetermined range are sampled as the object. In thisway, an object can be determined by its chromaticity, and the object maybe sampled without depending on the lightness or darkness of the object.

On the other hand as an example of image processing not concerned onlywith the object, the aforesaid evaluation unit comprises an evaluationcriterion wherein picture elements of the aforesaid image data areuniformly sampled and summed so as to obtain a feature amount, and inthe aforesaid processing unit, in one processing method, the featureamount obtained by the aforesaid uniform sampling is used when anaverage feature amount of the photographic image is used. In this case,when image data is converted based on the average feature amount in theaforesaid processing unit, the feature amount is obtained by uniformlysampling picture elements of the image data according to predeterminedevaluation criteria. Of course, the summation may be performed on allpicture elements of the photographic image, but it may be said that isno advantage as the processing amount increases. Hence, it is convenientto perform image processing based on the average feature amount of thephotographic image, e.g. saturation correction.

As another example of image processing which is not concerned only withthe object, the aforesaid evaluation unit comprises an evaluationcriterion wherein picture elements of the aforesaid image data areuniformly sampled and summed to obtain a feature amount, and in theaforesaid processing unit, in one image processing method, the featureamount obtained by uniform sampling is used when the edges of a featureamount distribution of the photographic image are used. In this case, inthe aforesaid processing unit, it is assumed that the ends of thefeature amount distribution obtained in the aforesaid evaluation unitare used. For example, to increase the contrast, image processing isperformed to find the luminance distribution, and the edges of thisluminance distribution are widened, but if the luminance distribution ofthe object were used in this case, other highlighted parts appear white.Therefore in this case, in the aforesaid evaluation unit, pictureelements of image data are uniformly sampled according to predeterminedcriteria, and summed to obtain the feature amount. This is suitable forimage processing using the edges of a feature amount distribution in anactual photographic image, e.g. for increasing contrast.

In the above, a continuous sequence of processes is performed comprisingpredetermined analysis of the image and image processing by specifyingimage processing indicators, but the analysis result itself is alsouseful.

It is a further object of this invention to provide an image evaluationdevice wherein it is easier to use an image evaluation result which isan analysis result done.

In the image evaluation device offered by this invention, photographicimage data comprising dot matrix picture elements is input, the imagedata for all picture elements is summed according to predeterminedcriteria, and is the image evaluation device which is based summationresult, and evaluate image, and the image is evaluated based on thesummation results. There are plural evaluation criteria for theaforesaid summation results, and the evaluation results are combinedwith a predetermined weighting based on these evaluation criteria.

According to the invention as thus comprised, the evaluation methodassumes that photographic image data comprising dot matrix pictureelements is input, the image data is summed for picture elements, andthe image is evaluated based on the summation results. Herein, there areplural evaluation criteria for these summation results, and theevaluation results are combined with a predetermined weighting based onthe evaluation criteria.

In other words, although some evaluation criteria are suitable forevaluating images where a sharp image is the object such as in the caseof portrait, other criteria are suitable for evaluating images where thebackground is the important object. A general evaluation may be made bysuitably combining plural evaluation criteria in parallel and varyingthe weightings.

As described hereabove, as this invention gives a general evaluation byvarying the weightings of plural evaluation criteria, it provides animage evaluating device which can be flexibly adapted to image featuredetermination.

Naturally, the concept of this invention for image evaluation by theabove techniques comprises many forms. Specifically, it compriseshardware and software, various modifications being possible as may beconvenient. When the concept of the invention is implemented by imageprocessing software, there naturally exist recording media on which thesoftware is recorded which can be used to perform the function of theinvention. Moreover, these image evaluating techniques may be applied toan image evaluating device and its system running on software media.

Various techniques may be employed to achieve the same object inapplying plural evaluation criteria to summation results. For example,all picture elements may be summed by weighting with differentevaluation criteria, but it will be appreciated a large amount ofprocessing is involved when the summation is applied to all pictureelements. Hence, as described above, the picture elements are firstsampled based on plural evaluation criteria, summed, and the summationresults combined with a predetermined weighting. In this case, the imagedata are sampled prior to summation, plural criteria are used by varyingthe criteria applied to the sampling, and the weighting of the summationresults is then adjusted before combination. An evaluation can thereforebe made with different weightings on the results based on pluralevaluation criteria. Due to this sampling of image data, pluralevaluation criteria may be employed according to the sampling method.

As one evaluation criterion, the data may of course be sampled uniformlyand summed. In this case, the image data is uniformly thinned and thewhole image is considered, which makes this a suitable criterion fordetermining scenic photographs, etc. In this way an optimum criterioncan be used while reducing the processing amount.

As an example of a criterion which can be applied whether or notsampling is used, is the evaluation of picture elements which have alarge variation relative to adjacent picture elements with a heavierweighting. In this case, the image variation of picture elements fromadjacent elements is detected, and clear image parts with a largevariation are given a heavier weighting in the summation. If this isdone, as image parts with a large variation are often parts of thephotograph which are clearly in focus, an image evaluation which weightsimportant parts of the image can be performed.

This criterion places more emphasis on the sharp parts of the image, andit is therefore naturally suitable for the determination of humanimages. Herein, image parts with a large variation may be evaluatedeither by introducing weighting as the picture elements are summed, orby summing only picture elements with a large variation. The weightingused in the evaluation is not necessarily fixed, but may also be allowedto vary according to the criterion. In this case, by varying theweighting for different evaluation criteria, an overall evaluationresult for the image can be deduced. Moreover various approaches arepossible, e.g. weightings may be varied individually so as to generateplural combinations which are then selected. In this way, by modifyingthe weighting for plural criteria, a more flexible evaluation can bemade.

Instead of an operator varying the weighting, this can be done based onthe image data itself. As an example of this, the weighting of theevaluation results may be varied based on the criteria. In this case,results are obtained according to various criteria, and the weighting ismodified in view of the suitability of the criteria in the light of theresults. As the results are used to vary the weighting, the workinvolved in the evaluation is less.

Various techniques may also be used to modify the weighting of thecriteria using the results. For example, if it is determined whether ornot the image data for picture elements should be sampled according toone criterion, the number of these picture elements may be taken as acriterion and the weighting increased when the number of pictureelements is large.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image processing system in which animage processing apparatus according to one embodiment of this inventionis applied.

FIG. 2 is a block diagram of the actual hardware of the image processingapparatus.

FIG. 3 is a block diagram showing another application example of theimage processing apparatus of this invention

FIG. 4 is a block diagram showing another application example of theimage processing apparatus of this invention.

FIG. 5 is a flowchart showing a front stage of main processing in animage processing apparatus according to this invention.

FIG. 6 is an illustration showing a case where an image variation amountis expressed by component values in orthogonal coordinates.

FIG. 7 is an illustration showing a case where an image variation amountis expressed by a differential value in adjacent picture elements in avertical axis direction and a horizontal axis direction.

FIG. 8 is an illustration showing a case when an image variation iscalculated between adjacent picture elements.

FIG. 9 is a diagram showing a region wherein a threshold value isvaried.

FIG. 10 is a flowchart for automatic division of regions.

FIG. 11 is a diagram showing a region setting.

FIG. 12 is a diagram showing this region setting in a modified example.

FIG. 13 is a flowchart showing the latter half of main processing.

FIG. 14 is a diagram showing the edges obtained by luminancedistribution edge processing and edge processing.

FIG. 15 is a diagram showing widening of a luminance distribution and areproducible luminance range.

FIG. 16 is a diagram showing a conversion table for widening theluminance distribution.

FIG. 17 is a diagram showing the general concept of brightening by γcorrection.

FIG. 18 is a diagram showing the general concept of darkening by γcorrection.

FIG. 19 is a diagram showing a correspondence relation for luminancemodified by γ correction.

FIG. 20 is a flowchart showing a case when saturation is emphasized inthe latter part of main processing.

FIG. 21 is a schematic view of a summation state of a saturationdistribution.

FIG. 22 is a diagram showing a relation between a saturation A andsaturation emphasis index S.

FIG. 23 is a flowchart showing a case when edges are emphasized in thelatter part of main processing.

FIG. 24 is a diagram showing the magnitude of image data and a statewhere image data to be processed is displaced.

FIG. 25 is a diagram showing a 5×5 picture element unsharp mask.

FIG. 26 is a block diagram of an image processing system in which animage processing apparatus according to one embodiment of this inventionis applied.

FIG. 27 is a flowchart showing image processing in the image processingapparatus according to this invention.

FIG. 28 is a diagram showing a sampling frequency.

FIG. 29 is a diagram showing a picture element sampling number.

FIGS. 30( a)-(c) are diagrams showing a relation between an image to beconverted and picture elements for sampling.

FIG. 31 is a diagram showing a block arrangement resulting from imagedivision.

FIG. 32 is a diagram showing a block luminance distribution.

FIG. 33 is a diagram showing an example of block weighting.

FIG. 34 is a diagram showing another example of block weighting.

FIG. 35 is a flowchart showing a case when a feature amount isreevaluated based on an edginess amount.

FIG. 36 is a diagram showing a relation between a main picture elementand edge picture elements for determining the edginess amount.

FIGS. 37( a)-(f) are diagrams showing an example of a filter forcomputing the edginess amount.

FIG. 38 is a flowchart showing a case when a feature amount isreevaluated based on an edginess amount.

FIG. 39 is an example of a photographic image.

FIGS. 40( a)-(d) are diagrams of a luminance distribution of aphotographic image photographed at night.

FIG. 41 is a block diagram of an image processing system in which animage processing apparatus according to one embodiment of this inventionis applied.

FIG. 42 is a flowchart showing a sampling part in an image processingapparatus according to this invention.

FIG. 43 is a flowchart showing a feature amount acquiring part and animage processing part.

FIG. 44 is a block diagram of an image processing system in which animage processing apparatus according to one embodiment of this inventionis applied.

FIG. 45 is a flowchart showing an image processing part in an imageprocessing apparatus according to this invention.

FIG. 46 is a block diagram showing an image evaluation option inputscreen.

FIG. 47 is a diagram showing how individual sampling results areconverted to weightings and combined.

FIG. 48 is a flowchart showing the latter stage of image evaluation andan image processing part.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Some preferred embodiments of this invention will now be described.

Embodiment 1

First, a description will be given of one form of image processorcomprising an image processing indicator specifying unit which comprisesan object determining unit, and automatically determines an object aspicture elements having a large variation. Conventionally, thisdetermination had to be performed by a human operator.

FIG. 1 shows a block diagram of an image processing system to which animage processing apparatus according to one embodiment of this inventionis applied. FIG. 2 shows an example of the actual hardware constructionby a schematic block diagram.

In FIG. 1, an image reader 10 outputs photographic image data whichrepresented photographs as dot matrix picture elements to an imageprocessing apparatus 20A. The image processing apparatus 20A determinesthe contents and extent of image processing, and then performs theprocessing. The image processing apparatus 20A outputs the processedimage data to the image output apparatus 30, and the image outputapparatus 30 outputs the processed image as dot matrix picture elements.From the image data output by the image processing apparatus 20A, animage variation amount is found for each picture element, and pictureelements having a large variation amount are determined to be those ofthe object. The content and extent of image processing are determined,and image processing is performed in line with this object image data.Therefore, the image processing apparatus 20A comprises an objectdetermining unit which finds an image variation amount in each pictureelement and determines picture elements having a large variation amountas the object, and processing unit which determines the content andextent of image processing in line with object image data.

A scanner 11 in FIG. 2 and digital still camera 12 or video camera 14correspond to a specific example of the image reader 10, the computersystem corresponds to a specific example of the image processingapparatus 20A comprising a computer 21, hard disk 22, keyboard 23,CD-ROM drive 24, floppy disk drive 25 and modem 26, and the printer 31and display 32 correspond to specific examples of the image outputapparatus 30. In case of this embodiment, the object is found to performappropriate image processing, so photographic data such as photographsare suitable as image data. A modem 26 is connected to the publictelecommunication line, and to an external network via the publictelecommunication line through which software and data can bedownloaded.

According to this embodiment, the scanner 11 and digital still camera 12which function as the image reader 10 output RGB (red, green, blue)gradation data. The printer 31 which is the image output apparatus 30requires input of CMY (cyan, magenta, yellow) or CMYK (to which black isadded) as gradation data, and the display 32 requires RGB gradation dataas input. Also, an operating system 21 a runs on the computer 21. Aprinter driver 21 b corresponding to the printer 31 and display driver21 c for the display 32 are built in. Processing is controlled by theoperating system 21 a, and an image processing application 21 d performspredetermined image processing together with the printer driver 21 b anddisplay driver 21 c when necessary. Therefore, the specific role of thiscomputer 21 which functions as the image processing apparatus 20A is toinput RGB gradation data, generate RGB gradation data for optimum imageprocessing, display the data on the display 32 via the display driver 21c, convert the data to CMY (or CMYK) binary data via the printer driver21 b, and print it on the printer 31.

In this way, according to this embodiment of the invention, a computeris interposed between the input-output apparatus to perform imageprocessing, but a computer is not absolutely necessary if the system iscapable of performing various types of image processing on image data.For example the system may be such that the image processing apparatuswhich determines the object and performs image processing is built intothe digital camera 12 a as shown in FIG. 3. The image is displayed on adisplay 32 a and printed by a printer 31 a using converted image data.Alternatively, the object is determined and image processing isperformed automatically from image data input via a scanner 11 b anddigital still camera 12 b or modem 26 b, as shown in FIG. 4.

The aforesaid determination of the object and image processing areperformed by an image processing program corresponding to a flowchartshown by FIG. 5 which is built into the computer 21. In the flowchartshown in the figure, it is determined whether or not the image is thatof the object.

According to this invention, picture elements for which the image issharp are determined to be those of the object based on the experimentalfact that the image is sharper for the object than for other parts. Whenimage data comprises dot matrix picture elements, gradation data isdisplayed showing RGB luminance for each picture element, and adifference amount between data for adjacent picture elements becomeslarge at the edge of the image. This difference amount is a luminancegradient, and is referred to as edginess. In a step SA110, the edginessof each picture element is determined. When the XY orthogonal coordinatesystem is considered in FIG. 6, vectors of the image variation amountmay be computed if the X-axis direction component and Y axis directioncomponent are found respectively. For a digital image comprising dotmatrix picture elements, assume that there are adjacent picture elementsin the vertical axis direction and horizontal axis direction as shown inFIG. 7, and assume that the luminance is expressed as f (x, y). In thiscase f(x, y) is R(x, y), G(x, y), B(x, y) which is the luminance of eachof the colors RGB, or it may be a total luminance Y (x, y). Strictlyspeaking, the relation between R(x, y), G (x, y), B (x, y) which is theluminance of each of the colors RGB, and total luminance Y(x, y), cannotbe converted without referring to color conversion charts, but a simplecorrespondence relation can be utilized as described hereafter. As shownin FIG. 7, a difference amount value fx in the X direction and adifference amount value fy in the Y direction may be written:fx=f(x+1,y)−f(x,y)  (1)fy=f(x,y+1)−f(x,y)  (2)Therefore the magnitude of the vector |g (X, y)| having thesedifferences as components may be written as:|g(x,y)|=(fx**2+fy**2)**(½)  (3)Edginess is of course represented by |g(x, y)|. The picture elements areactually arranged in length and breadth as a grid shown in FIG. 8, therebeing eight picture elements in the center. Therefore, expressing adifference amount of image data between adjacent picture elements as avector, the sum of this vector may be taken as the image variationamount.

As the edginess may be found for each picture element in this way,picture elements having a large edginess when compared with a thresholdvalue may be determined to be object picture elements. However fromactual experience, the object is often situated in the center of thefigure. This fact provides proof that the use of an arrangement where alarge number of picture elements are sampled from the center area forimage processing, gives satisfactory results.

For this reason, the threshold values Th1, Th2, Th3 used for comparisonin each part of the center of the image are arranged to be different asshown in FIG. 9. In this example, of course, the relationTh1<Th2<Th3  (4)holds. The threshold value is lower the nearer the center, and this areais determined to be the object even if the edginess is relatively low.

As the threshold value varies as shown in the figure, the area isdivided uniformly into three equal parts in the horizontal and verticaldirections from the center of the image. In a step SA120, the thresholdvalue used for comparison is determined based on the area in which thepicture element used for edginess determination is located. Thethreshold value is compared with the edginess in a step SA130, and it isdetermined whether or not the variation amount is large. As a result ofthis comparison, if the edginess is large, it is determined that thispicture element is a picture element of the object, and image data ofthe picture element is stored in a work area in a step SA140. The workarea may be a RAM in the computer 21, or the hard disk 22.

The above processing is performed for each picture element of imagedata. In a step SA150, the picture element to be processed is displaced,and processing is repeated until it is determined that processing hasbeen completed in a step SA160.

In the embodiment described above, the area division for modifying thethreshold value was always based on the central part of the image, butthe area division may also be varied based on the edginess distribution.FIG. 10 is a flowchart for suitably varying the area division, and FIG.11 shows the areas so divided.

In this case also, subsequent processing is performed on each pictureelement while moving the picture element to be processed in the same wayas above. After the aforesaid edginess was determined in a step SA210,it is summed in the horizontal axis direction in a step SA220, andsummed in the vertical axis direction in a step SA230. Picture elementsto be processed are displaced in a step SA240, and the process loopsuntil it is determined in a step SA250 that processing for all pictureelements is complete.

After summation is completed for the horizontal axis direction andvertical axis direction, a maximum distribution position on thehorizontal axis is determined in a step SA260, and a maximumdistribution position on the vertical axis is determined in a stepSA270. As shown in FIG. 11, a high edginess part along the horizontaland vertical axis direction is regarded as the center of the image, thearea being divided as follows.

The distance to an end from the center is divided in half in thevertical direction and the horizontal direction. The threshold value isTh1 for the inside, the remaining distance being divided in half with athreshold value Th2 for the inner area and a threshold value Th3 for theouter area. Ina step SA280, by dividing the area as described above,comparison criteria are determined, and in a step SA290, object pictureelements are determined by perform sampling based on edginess by thesame processing as that of the steps SA110-SA160 mentioned aboveaccording to the correspondence between this area and threshold value.

In this example, after finding the center area, the area was dividedinto two equal parts in the directions of both the horizontal axis andvertical axis, but the area may also be divided in other ways, e.g.based on the edginess distribution. The actual area division may besuitably varied.

For example, in the example stated above, summation in the horizontalaxis direction and vertical axis direction were performed in pictureelement units, but the image may be divided into a relatively largergrid as shown in FIG. 12. The summation may then be made in these gridunits, the position of the maximum distribution determined, and areadivision performed.

If object picture elements can be sampled in this way, the optimum imageprocessing can be determined and performed based on image data for thesepicture elements. FIG. 13 is a flowchart showing increase of contrastand lightness compensation as an example.

In the basic technique to increase contrast according to thisembodiment, a luminance distribution is found based on object imagedata, and if this luminance distribution uses only part of the originalgradation (255 gradations), the distribution is expanded.

Therefore, a histogram of luminance distribution is generated in a stepSA310, and an expansion width is determined in a step SA320. When theexpansion width is determined, both ends of the luminance distributionare found. A luminance distribution of a photographic image is generallylike a hump as shown in FIG. 14. The distribution may of course havevarious positions and shapes. The width of the luminance distribution isdetermined by where the two ends are located, but the points where thedistribution number is “0” where the distribution slopes away cannot betaken as the ends. There is a case where the distribution number variesin the vicinity of “0” in the lower sloping part. This is because from astatistical viewpoint, it changes without limit as it approaches “0”.

Therefore, the two ends of the distribution are taken to be a part inthe distribution shifted somewhere towards the inside by a certaindistribution fraction from the brightest side and the least bright side.In the area in this embodiment, this distribution fraction is set toO.5%, but the fraction may be modified as deemed appropriate. In thisway, white spots and black spots due to noise can also be ignored bycutting the upper and lower ends by a certain distribution fraction.Specifically, if such processing is not performed and there are whitespots or black spots, these become the two ends of the luminancedistribution. In a 255 gradation luminance value distribution, the lowerend is usually “0” and the upper limit is “255”, but the above problemis avoided by considering the end of the distribution to be a pointsituated at 0.5% from the end in terms of picture elements.

In the actual processing, 0.5% of the number of picture elements sampledas the object is computed, the distribution numbers are summed towardsthe inside in sequence from the luminance value at the upper end and theluminance value at the lower end in a reproducible luminancedistribution, and the luminance value corresponding to 0.5% isdetermined? Hereafter, the upper limit will be referred to as ymax andthe lower limit as ymin.

When the reproducible range of luminance is “0”-“255”, the luminance Yconverted from the luminance y before conversion, and the maximum valueymax and minimum value ymin of the luminance distribution, is given bythe following equations:Y=ay+b  (5)where a=255/(ymax−ymin)  (6)b=−a·ymin or 255−a·ymax  (7)

In the above equation, when Y<0, Y is set equal to 0, and when Y>255, Yis set equal to 255. a is a slope and b is an offset. According to thisconversion equation, a luminance distribution having a certain narrowwidth can be enlarged to a reproducible range, as shown in FIG. 15.However, when the reproducible range was increased to the maximum toexpand the luminance distribution, highlighted areas are white and highshadow areas come out black. To prevent this, according to thisembodiment, the reproducible range is limited, i.e. a luminance value of“5” is left as a range which is not expanded at the upper and lower endsof the reproducible range. As a result, the parameters of the conversionequation are given by the following equations:A=245/(ymax−ymin)  (8)B=5−a/ymin or 250−a/ymax  (9)

In this case, in the ranges y<ymin and y>ymax, conversion is notperformed.

However, if this expansion factor (corresponding to a) is applied, avery large expansion factor may be obtained. For example at dusk,although the width of contrast from the brightest to the darkest part isnaturally narrow, if the contrast of such an image were considerablyincreased, it would appear to be converted to a daytime image. As such aconversion is not desired, a limit is imposed on the increase factor sothat it is equal to or greater than 1.5 (−2). Due to this, duskcorrectly appears as dusk. In this case, processing is performed so thatthe center position of the luminance distribution does not vary.

However, in luminance conversion, it is unreasonable to perform theaforesaid conversion (Y=ay+b) on every occasion. This is because theluminance y can only lie within the range “0” to “255”, and theluminance Y after conversion can be found for all possible values of ybeforehand. These can therefore be stored in a table such as is shown inFIG. 16.

This conversion table corresponds to the expansion width determinationprocessing of the step SA320, and it allows image data to be converted.However, as it is very useful not only to emphasize contrast byincreasing the luminance range but also to adjust luminance at the sametime, the luminance of the image is determined in a step SA330 and acorrection parameter is generated.

For example, the hump of the luminance distribution may be displaced tothe side which is brighter overall as shown by the dotted line in FIG.17 when the hump is nearer dark as shown by the solid line in thefigure. Conversely, the hump of the luminance distribution may bedisplaced to the side which is darker overall as shown by the dottedline in FIG. 18 when the hump is nearer bright on the whole as shown bythe solid line in the figure.

By performing various experiments, according to this embodiment, aMedian ymed in the luminance distribution is found. When the median ymedis less than “85”, the image is determined to be dark, and is lightenedby a γ correction corresponding to the following γ value:γ=ymed/85  (10)orγ=(ymed/85)**(½)  (11).

In this case even if γ<0.7, γ is set equal to 0.7. If such a limit isnot provided, a night scene appears as if it is in daylight. If theimage is made too light, it becomes too white and contrast tends to betoo low, hence it is preferable to perform processing such as emphasisin conjunction with saturation.

On the other hand, when the median ymed is greater than “128”, the imageis determined to be a light image and is darkened by a γ correctioncorresponding to the following γ value:γ=ymed/128  (12) orγ=(ymed/128)**(½)  (13).In this case even if γ>1.3, γ is set equal to 1.3 so that the image doesnot become too dark.

This γ correction may be applied to the luminance distribution beforeconversion, or to the luminance distribution after conversion.Correspondence relations for γ correction are shown in FIG. 19. Whenγ<1, the curve bulges upwards, whereas when γ>1, the curve bulgesdownwards. Of course, the result of this γ may also be reflected in thetable shown in FIG. 16, and the same correction may be applied to tabledata.

Finally, in a step SA340, it is determined whether or not contrastcorrection and lightness compensation are necessary. In thisdetermination, the aforesaid expansion factor (a) and γ value arecompared with suitable threshold values, and when the expansion factoris large and the γ value exceeds a predetermined range, it is determinedthat such correction is necessary. If it is determined to be necessary,conversion of image data is performed. Specifically, the need for imageprocessing and its extent are assessed in the steps SA310-SA340, and therequired image processing is performed in the a SA350. The processingunit to accomplish this comprises hardware and software.

Conversion on the basis of equation (5) is performed when it isdetermined that image processing is necessary. This equation may also beapplied to correspondence relations between RGB component values. Thecomponent values after conversion (R, G, B) relative to the componentvalues before conversion (R0, G0, B0) may be found from:R=a·R0+b  (14)G=a·G0+b  (15)B=a·B0+b  (16)Herein, the RGB component values (R0, G0, B0), (R, G, B) have the samerange when the luminance y, Y has the gradation “0” “255”, so theaforesaid conversion table of luminance y, Y may be used withoutmodification.

Therefore, the conversion table corresponding to equations (14)-(16) isreferred to for the image data (R0, G0, B0) for all picture elements ina step SA350, and the process for obtaining the image data (R, G, B)after conversion is repeated.

In this processing unit, the determination is performed only forcontrast correction and lightness compensation, but specific examples ofimage processing are not limited to this.

FIG. 20 shows a flowchart for performing image processing for saturationemphasis.

First, if the object and determined picture element data have saturationas a component element, a distribution may be found using saturationvalues. However, as the data comprises only RGB component values,saturation values cannot be obtained unless they are converted to acolor specification space which comprises direct component values? Forexample, in a Luv space which is a colorimetric system, the L axisrepresents luminance (lightness), and hues are represented by the U axisand V axis. Herein, as the distance from the intersection point of the Uaxis and V axis shows saturation, the saturation is effectively(U**2+V**2)**(½).

Such a color conversion between different color specification spacesrequires an interpolation to be performed while referring to a colorconversion table which stores correspondence relationships, and thecomputation amount is enormous. In view of this, according to thisembodiment, standard RGB gradation data is utilized directly as imagedata, and saturation substitute values X are found as follows:X=|G+B−2×R|  (17)

Actually, the saturation is “0” when R=G=B, and is a maximum valueeither for any of the single colors RGB or for a mixture of two colorsin a predetermined proportion. Due to this, it is possible toappropriately represent saturation directly. For yellow which is amixture of the color red with green and blue, from the simple equation(17), the maximum saturation value is obtained, and when the componentsare equal, this is “0”. For green or blue alone, about half the maximumvalue is attained. Of course, substitutions may be made using theequations:X′=|R+B−2×G|  (18)X″=|G+R−2×B|  (19)

In a step SA410, a histogram distribution is found for the saturationsubstitution value X. In equation (17), saturation is distributed in therange of minimum value “0”-maximum “511”, and the distribution obtainedis approximately as shown in FIG. 21. In a next step SA420, based on thesummed saturation distribution, a saturation index is determined forthis image. According to this embodiment, a range occupied by the upper16% of distribution number is found within the number of pictureelements determined to be the object. Assuming that the lowestsaturation “A” in this range represents the saturation of the image, asaturation emphasis index S is determined based on the followingequation. In other words, it is assumed that:If A<92,S=−A×(10/92)+50  (20)If 92≦A<184,S=−A×(10/46)+60  (21)If 184≦A<230,S=−A×(10/23)+100  (22)If 230≦A,S=0  (23)

FIG. 22 shows a relation between this saturation “A” and the saturationemphasis index S. As shown in the figure, in the range between themaximum value “50”-minimum value “0”, the saturation index S graduallyvaries so that it is large when the saturation A is small, and smallwhen the saturation A is large.

When saturation is emphasized based on the saturation emphasis index S,if the image data is provided with saturation parameters as statedabove, the parameters may be converted. When an RGB color specificationspace is adopted, the data must first be converted into the Luv systemwhich is a standard color system, and moved in a radial direction withinthe Luv space. However, this means that RGB image data must first beconverted into image data in Luv space, and then returned to RGB aftersaturation emphasis which involved an enormous amount of computation.Therefore, RGB gradation data are used without modification forsaturation emphasis.

When the components are component values of hue components which are ina schematic pair relation as in the case of the RGB color specificationspace, the color is grey and there is no saturation if R=G=B. Thereforeif the component which has the minimum value in RGB is considered merelyto have a reduced saturation without having any effect on the hue of thepicture elements, the minimum of each component may be subtracted fromall the component values, and the saturation emphasized by increasingthe value of the difference.

First, a parameter Sratio which is useful for calculation is found fromthe aforesaid saturation emphasis index S by the equation:Sratio=(S+100)/100  (24)

In this case the saturation emphasis parameter Sratio=1 when thesaturation emphasis index S=0, and saturation is not emphasized. Next,assuming that the value of the blue (B) component in the components (R,G, B) of the RGB gradation data is the minimum, this saturation emphasisparameter Sratio is used to perform the following conversion:R′=B+(R−B)×Sratio  (25)G′=B+(G−B)×Sratio  (26)B′=B  (27)

As a result, two-way color conversions between the RGB colorspecification space and the Luv space are rendered unnecessary, andcomputing time can be reduced. In this embodiment, as for non-saturationcomponents, the component with the minimum value was simply subtractedfrom other component values, but other conversion equations may be usedto subtract non-saturation components. However when only the minimumvalues are subtracted as in equations (25)-(27), there are nomultiplications or divisions so the computation is easier.

When the equations (25)-(27) are used, a good conversion is possible,however in this case when saturation is emphasized, luminance alsoincreases so the image becomes lighter overall. Therefore, theconversion is performed using a difference value obtained by subtractingan equivalent luminance value from each component value.

Firstly to find the luminance, as the computation becomes bulky when acolor conversion is performed in Luv space, the following equation whichis used in television for example, is used to find the luminance fromRGB. The luminance Y is given byY=0.30R+0.59G+0.11B  (28).

It will be assumed that saturation emphasis is given byR′=R+ΔR  (29)G′=G+ΔG  (30)B′=B+ΔB  (31).

These addition/subtraction values ΔR, ΔG, ΔB are found by the followingequations based on difference amount values:ΔR=(R−Y)×Sratio  (32)ΔG=(G−Y)×Sratio  (33)ΔB=(B−Y)×Sratio  (34)

As a result, the conversion can be performed byR′=R+(R−Y)×Sratio  (35)G′=G+(G−Y)×Sratio  (36)B′=B+(B−Y)×Sratio  (37)

Conservation of luminance is clear from the following equations:

$\begin{matrix}{Y^{\prime} = {Y + {\Delta\; Y}}} & (38) \\\begin{matrix}{{\Delta\; Y} = {{0.30\Delta\; R} + {O{.59}\Delta\; G} + {0.11\Delta\; B}}} \\{= {{Sratio}\;\left\{ {\left( {{0.30\; R} + {0.59\; G} + {0.11\; B}} \right) - Y} \right.}} \\{= O}\end{matrix} & (39)\end{matrix}$Also, when the input is grey (R=G=B), the luminance Y=R=G=B, theaddition/subtraction values ΔR=ΔG=ΔB=0 and there is no color in theachromaticity. If equations (35)-(37) are utilized, the luminance isstored, and the image does not become lighter overall even if saturationis emphasized.

If the saturation emphasis index Sratio is found as described above, itis compared with a predetermined threshold value in a step SA430, and itis determined whether the image requires saturation emphasis. If it isnecessary, the image data for all picture elements is then convertedbased on equations (35)-(37) in a step SA440.

Therefore, in the steps SA410-SA430, the need for saturation emphasisprocessing and its extent is determined, and when it is deemed necessaryin the step SA430, saturation emphasis processing is performed. Aprocessing unit therefore comprises hardware and software to accomplishthese functions.

To determine the content and extent of image processing based on objectpicture elements, edge emphasis processing may also be used. FIG. 23shows a flowchart of this edge emphasis processing. As object pictureelements are selected, the edginess is averaged for object pictureelements by dividing the integrated edginess by the number of pictureelements in a step SA510. If the number of picture elements is E (I)Pix, the sharpness degree SL of the object image may be computed by

$\begin{matrix}{{SL} = {\sum\limits_{x,y}\;{{{{g\left( {x,y} \right)}}/{E(I)}}{{pix}.}}}} & (40)\end{matrix}$

In this case, the degree of sharpness is lower (the image appears moreblurred) the lower the SL value of the image, and the degree ofsharpness is higher (the image appears clearer) the higher the SL valueof the image.

On the other hand because the sharpness of the image is subjective, thedegree of sharpness SL is found in the same way for image data having anoptimum sharpness obtained experimentally, this value is set as an idealsharpness SLopt, and a degree of edge emphasis Eenhance is found fromthe relationEenhance=ks·(SLopt−SL)**(½)  (41)in a step SA520.Herein, ks varies based on the magnitude of the image, and when theimage data comprises height dots and width dots in the vertical andhorizontal directions as shown in FIG. 24, ks is found fromks=min(height,width)/A  (42)Herein, min (height, width) denotes the smaller of height and width, andA is the constant “768”. It will be understood that these relations areobtained from experiment, and may be modified as appropriate. However,good results are basically obtained by making the degree of emphasislarger the larger the image.

When the degree of edge emphasis Eenhance is found in this manner, it iscompared with a predetermined threshold value in a step SA530, and it isdetermined whether edge emphasis is necessary. If it is deemed to benecessary, edge emphasis processing is performed on all picture elementsin a step SA540.

In edge emphasis processing, a luminance Y′ after emphasis relative tothe luminance Y of each picture element before emphasis is computed by:Y′=Y+Eenhance·(Y−Yunsharp)  (43)Herein, Yunsharp is unsharp mask processing relative to image data ofeach picture element. Unsharp mask processing will now be described.FIG. 25 shows an example of a 5×5 picture element unsharp mask. In thisunsharp mask 41, the central value “100” is a weighting of a pictureelement Y (x,y) in matrix image data, which is used for multiplicationwith weighting corresponding to a numerical value in the grid of themask for edge picture elements. When this unsharp mask 41 is utilized,multiplication is performed based on the computational equation

$\begin{matrix}{{{Yunsharp}\left( {x,y} \right)} = {\left( {1/396} \right){\sum\limits_{i,j}\;\left( {{Mij} \times {Y\left( {{x + 1},{y + j}} \right)}} \right)}}} & (44)\end{matrix}$In equation (44), “396” is a total value of weighting coefficients, andit is the total value of each grid division in unsharp masks ofdifferent size. Mij is a weighting coefficient written in a griddivision of the unsharp mask, and Y (x, y) is image data for eachpicture element. ij is expressed in horizontal and vertical coordinatevalues for the unsharp mask 41.

The meaning of the edge emphasis computation based on equation (43) isas follows. As Yunsharp (x, y) is added by making the weighting of edgepicture elements lower than that of main picture elements, the result is“unsharp” image data. Images which are made unsharp in this way have thesame meaning as those subjected to a low pass filter. Therefore, “Y(x,y)−Unsharp(x,y)” means low frequency components are removed from thetotal components, which has the same meaning as applying a high passfilter. If high frequency components which passed through the high passfilter are multiplied by the edge emphasis Eenhance and added to “Y(x,y)”, the high frequency components are increased in direct proportion tothe edge emphasis Eenhance, and the edges are thereby emphasized.Considering the situation when edge emphasis is needed, edge emphasisrefers to the edge of the image, and it therefore may be computed onlywhen there is a large difference of image data between adjacent pictureelements. If this is done, there is no need to compute an unsharp maskfor most image data which is not edge parts, so the amount of processingis vastly reduced.

In the actual computation, if we write:Δ=Y−Y′  (45)from the luminance Y′ after emphasis and the luminance Y beforeemphasis, R′ G′ B′ after conversion may be computed as:R′=R+deltaG′=G+deltaB′=B+delta  (46).

Therefore, in this edge emphasis processing in the steps SA510-SA530,the need for edge emphasis and its extent are determined, and imageprocessing is performed when it is determined to be necessary in thestep SA530. A processing unit comprising hardware and software isprovided to perform these functions.

It is determined whether to perform image processing regarding contrastcorrection, lightness compensation, saturation emphasis and edgeemphasis. However, it is not absolutely necessary to make a choice as towhether or not to perform image processing. Specifically, an emphasisdegree may be set for each, and image processing performed with the setemphasis degree. Of course, in this case also, the contents and extentof image processing which should be performed are determined, and theprocessing is performed.

Next, the operation of this embodiment having the aforesaid constructionwill be described.

A photographic image is read by the scanner 11, and printed by theprinter 31. Specifically, when the operating system 21 a is running onthe computer 21, the image processing application 21 d is started, andreading of the photograph is started by the scanner 11. After the readimage data has been assimilated by the image processing application 21 dvia the operating system 21 a, the picture elements to be processed areset in initial positions. Next, the edginess is determined based onequations (1)-(3) in the step SA110, and the edginess is compared with athreshold value in the step SA120. When the edginess is large, it isdetermined that the picture element to be processed is an edge pictureelement, and image data for the corresponding picture element is storedin the work area in the step SA130. In the step SA140, it is determinedwhether or not picture elements to be processed are to sampleduniformly, and if so, in the step SA150, image data for these pictureelements is stored in the work area. The above processing is repeateduntil it is determined to have been performed for all picture elementsin the step SA160 while picture elements to be processed are displacedin the step SA150.

When this has been performed for all picture elements, image data forpicture elements determined to be those of the object are stored in thework area. Therefore, even if the situation of a photographic image readfrom the image data in this work area is determined, the nature of theimage is not misinterpreted due to the effect of the background, etc.According to this embodiment, image data was stored in the work area,but from the viewpoints of memory capacity and processing time, it isnot absolutely necessary to store image data itself in the work area.Specifically, histograms of luminance distribution or saturationsubstitute value distribution for picture elements determined to bethose of the object are generated, and histogram information maytherefore be pre-stored in the step SA140.

When contrast correction and luminance compensation are performedautomatically, a histogram of luminance distribution is found in thestep SA140 or the step SA310, a parameter for expansion processing isdetermined based on equations (8), (9) in the step SA320, and aparameter for luminance compensation is determined based on equations(10)-(13) in the step SA330. These parameters are compared with apredetermined threshold value in the step SA340, and if it is determinedthat image processing should be performed, the luminance is convertedbased on the aforesaid parameter in the step SA350. In this case, theluminance conversion table shown in FIG. 16 may be generated to reducethe computing amount, and image data may be converted based on equations(14)-(16).

Subsequently, the processed image data may be displayed on the display32 via the display driver 21 c, and if it is satisfactory, it is printedby the printer 31 via the printer driver 21 b. In other words theprinter driver 21 b inputs RGB gradation data with emphasized edges,performs rasterization corresponding to the print head area of theprinter 31 after a predetermined mapping, color converts rasterized datafrom RGB to CMYK, converts CMYK gradation data to binary data, andoutputs it to the printer 31.

Due to the above processing, photographic image data read via thescanner 11 is automatically subjected to the optimum contrast correctionand lightness compensation, displayed by the display 32, and printed bythe printer 31. In other words, it is determined whether or not contrastcorrection and lightness compensation are necessary based on the objectpart of the corresponding photographic image, and if it is necessary,the optimum degree of image processing is performed. When object pictureelements are sampled, although edginess is required to determineparameters, it is not necessary to find the edginess for all pictureelements. The edginess may be determined for sampled picture elements,and a determination made as to whether or not the picture elements arethose of the object.

The invention is not limited to this contrast correction and luminancecompensation. In the case also of saturation emphasis and edge emphasis,picture elements for which there is a large variation amount aredetermined to be those of the object. The content and extent of imageprocessing are determined based on image data for object pictureelements, and the necessary processing is performed.

Hence, the computer 21 which is the core of image processing calculatesthe edginess, which is the image variation amount, from data foradjacent picture elements in the step SA110, selects only images whichhave a large edginess and determines them to be object picture elementsin the steps SA120, SA130, and calculates optimum parameters forperforming contrast correction and lightness compensation from imagedata for object picture elements in the steps SA310-SA330. An imageprocessing indicator can therefore be determined based on image data forobject picture elements, and optimum image processing can be performedautomatically.

Embodiment 2

Next, an embodiment of an image processing apparatus will be describedwherein an image processing indicator specifying unit comprises afeature amount uniform sampling unit, and a predetermined weighting isapplied after sampling without a large computational amount in thesampling stage so as to automatically perform optimum image processing.

FIG. 26 is a block diagram of an image processing system to which theimage processing apparatus according to one embodiment of this inventionis applied. The actual hardware construction may be similar to thesystem shown in FIG. 2.

In FIG. 26, an image reader 10 outputs photographic image datarepresented as dot matrix picture elements to an image processingapparatus 20B, and the image processing apparatus 20B performs imageprocessing after determining a degree of emphasis via a predeterminedprocess. The image processing apparatus 20B outputs the processed imagedata to an image output device 30, and the image output device 30outputs the processed image in dot matrix picture elements. Herein, theimage data which the image processing apparatus 20B outputs is obtainedby uniformly sampling feature amounts from picture elements by apredetermined criterion, reevaluating them with a predeterminedweighting, and processing them with an emphasis degree determinedaccording to the reevaluated feature amounts. Therefore, the imageprocessing apparatus 20B comprises a feature amount uniform samplingunit which uniformly samples feature amounts, a feature amount weightingreevaluation unit which reevaluates sampled feature amounts with apredetermined weighting, and a processing unit which performs imageprocessing with the degree of emphasis according to the reevaluatedfeature amounts.

Specifically, the determination of the object and accompanying imageprocessing are performed by an image processing program in the aforesaidcomputer 21 corresponding to a flowchart shown in FIG. 27. In theflowchart shown in the figure, image processing is performed to adjustthe contrast of the image. After sampling luminance which is a featureamount while uniformly thinning out picture elements from the wholeimage in a step SB110, this feature amount is reevaluated by applying apredetermined weighting in a step SB120, and image processing to adjustluminance is performed in steps SB130-SB160.

In the step SB110, the luminance of each picture element in dot matriximage data in the horizontal and vertical directions is found as shownin FIG. 24, and a histogram is generated. In this case, if theprocessing is applied to all picture elements it can be said to beprecise, but as the summation results are reevaluated by weighting, itdoes not necessarily have to be precise. Therefore picture elementswherefrom the luminance has been sampled to within the limits of acertain error are thinned out, processing amount is reduced, and theprocess is thereby speeded up. According to statistical error, an errorfor a sample number N can generally be represented by1/(N**(½))where ** represents involution.Therefore in order to perform processing with an error of around 1%,N=10000.

Herein, the bitmap screen shown in FIG. 24 is number of (width)×(height)picture elements, and a sampling period ratio is given byratio=min(width,height)/A+1  (47).min (width, height) is the smaller of width and height Herein, and A isa constant. The sampling period ratio mentioned here expresses howfrequently to perform sampling in numbers of picture elements, and themark O in FIG. 28 shows the case where the sampling period ratio=2. Inother words, one picture element is sampled every two picture elementsin the vertical and horizontal directions, so sampling is performedevery other picture element. The number of sampling picture elements in1 line when A=200 is as shown in FIG. 29.

As is clear from the figure, except for the case when the samplingperiod ratio=1 when sampling is not performed, at least 100 pictureelements are sampled

when there is a width of 200 picture elements or more.

Therefore, when there are 200 or more picture elements in the verticaland horizontal directions (100 picture elements)×(100 pictureelements)=10000 picture elements are sampled and the error is 1% orless.

The reason for taking min (width, height) as a basis is as follows. Forexample, as shown by the bitmap in FIG. 30( a), if width>>height and thesampling period ratio is determined by width which is the longerdirection, only two lines of picture elements, i.e. the top edge andbottom edge, can be sampled in the vertical direction as shown in (b).

However, if the sampling period ratio is determined based on the smallerof the two as min (width, height), thinning which includes the middlepart can be performed even in the lesser, vertical direction as shown in(c). In other words, sampling with a predetermined number of samplingscan be guaranteed.

Here, the feature which is sampled with thinning of picture elements isluminance. As described above, according to this embodiment, the datahandled by the computer 21 is RGB gradation data, and it does notdirectly have luminance values. To calculate luminance, a colorconversion to the Luv color specification space could be performed, butthis is not a good solution due to the problem of computing amount.

For this purpose, the aforesaid equation (28) which calculates luminancedirectly from RGB and is used for television, etc., is utilized here.

Also, the luminance histogram is not summed for the whole image, but theinput image is divided into 3 horizontal blocks and 5 vertical blocks,i.e. a total of 15 blocks, and the summation carried out for each block,as shown in FIG. 31. According to this embodiment there are 15 blocks,however there is of course no restriction on the block division used.

In particular, with printer drivers, etc., image data is received froman application in block units, and these blocks may also be used todemarcate areas for weighting.

The reason for summing in blocks in this way is to reduce the processingamount required. As reevaluation with weighting is performed in the stepSB120 it is not

absolutely necessary to perform the summation for each block, and thesummation may instead take the form of a histogram which considers theweighting for each selected picture element. Moreover, as the weightingchanges relative to the summation result regardless of blocks, thesummation can also be made with one histogram using a weightingdepending on the block. FIG. 32 is a figure showing an example ofluminance distribution of the block Bi.

When the summation is to performed for each block, a reevaluation ismade by weighting according to area in the step SB120. FIG. 33 and FIG.34 show examples of weighting each block. In the case of an ordinaryphotographic image, the object being photographed is usually in thecenter. In this sense, the feature amount should be evaluated placingmore on the center part of the image data. On the other hand, in thecase of a souvenir photo taken in front of a building, the person beingphotographed is generally in bottom center. Specifically, the person isusually in the lower part of the image with respect to height aboveground. In this case therefore, the feature amount should be evaluatedby weighting the lower middle part of the image. FIG. 33 shows anexample of the former case, and FIG. 34 shows an example of the latter.

If the weighting of each block=Wi (i) (i=1-15) and the weighted,reevaluated luminance distribution is DY,

$\begin{matrix}{{SP} = {\sum\limits_{i = {1 \sim 15}}\;{Wi}}} & (48) \\{{Ki} = {{Wi}/{SP}}} & (49)\end{matrix}$then:

$\begin{matrix}{{DY} = {\sum\limits_{i = {1 \sim 15}}\;{{Ki}*{dYi}}}} & (50)\end{matrix}$

After the histogram of the reevaluated luminance distribution isobtained in this way, the intensity of image processing is calculatedfrom this feature amount. In other words, the width for increasingcontrast is determined. This is done exactly as described above, i.e. amaximum value ymaX, minimum value ymin and median ymed are acquired inthe step SB130, the expansion factor a and offset b are found in thestep SB140, and processing to generate a conversion table is performedin the step SB150. In the step SB160, for image data (R0, G0, B0) forall picture elements, the processing to obtain image data (R, G, B)after conversion is repeated while referring to conversion tablescorresponding to equations (14)-(16).

A processing unit is provided comprising the hardware and software toperform the steps SB130-SB160.

It will moreover be understood that although according to thisembodiment, the image processing described is that of contrastcorrection and lightness compensation, the invention may be applied inexactly the same way to other image emphasis processing.

In the above processing, the feature amount was reevaluated by weightingaccording to a position in the image. However the weighting criterion isnot limited to this, and various other types of criterion are possible.As an example, FIG. 35 shows a flowchart for the case where the objectbeing photographed is detected from the image variation amount, and theweighting is varied.

A step SB210 replaces the above step SB110, and luminance is summedwhile thinning out picture elements uniformly. However, instead ofsummation only of luminance, edginess can also be summed as shown below.

In the input image it can be said that contrast of the backgroundchanges gradually; on the other hand, the object is sharp and so thereis an intense change of luminance. Therefore, a difference of density isfound between one picture element and surrounding picture elements asshown in FIG. 36. This density difference is taken as the edginess ofthe picture element considered. This density difference may be computedby applying a filter. FIG. 37 (a)-(f) show several examples of such afilter, and show weighting coefficients when the luminance is weightedfor a specific picture element and eight surrounding picture elements.Here, in the case of (a), the weighting is found for nine pictureelements, so nine multiplications and eight additions are necessary tocalculate the edginess of each picture element. When the image becomeslarge, it is impossible to ignore this computing amount, so fivemultiplications and four additions are performed in (b), (c), threemultiplications and two additions are performed in (d), (e) and oneaddition are performed in (f).

In these examples, the picture element in question is compared only withsurrounding picture elements. With the “unsharp mask”, the sharpness ofthe considered picture element may be found by using a wider range ofimage data. However, as edginess in this embodiment is only a tool toevaluate the weighting per block, even the filters of these examples ofreduced computation amount give sufficiently good results.

The summation of edginess may be performed by summing the edginess ofpicture elements for each block. Alternatively, when the absolute valueof this edginess is larger than a predetermined threshold valueabsolutely, the picture element is determined to be an edge pictureelement, and the total number of edge picture elements for each block issummed. When the edginess in each block is written as ERi (i=1-15), thistotal number SE is given by:

$\begin{matrix}{{SE} = {\sum\limits_{i = {1 \sim 15}}\;{ERi}}} & (51)\end{matrix}$so the weighting coefficient KEi itself may be expressed asKEi=ERi/SE  (52).Therefore the luminance distribution DY which is reevaluated byweighting may be calculated as:

$\begin{matrix}{{{DY} = {\sum\limits_{i = {1 \sim 15}}\;{{KEi}*{dYi}}}},} & (53)\end{matrix}$Also, if the total number of edge picture elements in each block is ENi(i=1-15), the total number SE is:

$\begin{matrix}{{SE} = {\sum\limits_{{i = 1},15}\;{{ENi}.}}} & (54)\end{matrix}$The weighting coefficient KEi itself is expressed asKEi=ENi/SE  (55),so the luminance distribution DY which was reevaluated by equation (53)can be obtained. In any case, the luminance distribution DY isreevaluated based on the computational equation (53) in the step SB220.

In this example, the luminance of picture elements determined to be edgepicture elements is not sampled. Instead, the edginess and total numberof edge picture elements is merely used for determining block weightingcoefficients. In other words, instead of summing the feature amounts forpicture elements having a specific property (edginess), an averagefeature amount which is not uneven can be obtained for the block.

If the luminance distribution is reevaluated in this way, the contrastmay be increased and the lightness may be modified by the processing ofthe aforesaid steps SB130-SB160.

In view of the fact that the original object is often that of a person,a reevaluation may also be made by placing more weight on pictureelements with skin color. FIG. 38 shows a flowchart wherein attention ispaid to a specific color to determine block weighting coefficients.

In a step SB310 corresponding to the step SB110, luminance is summed bythe same thinning process, it is determined whether or not pictureelements appear to have skin color based on the chromaticity of eachpicture element, and all picture elements having skin color are thensummed. For chromaticity, x-y chromaticity is calculated for eachpicture element. Now, ifr=R/(R+G+B)  (56)g=G/(R+G+B)  (57)when the RGB gradation data of object picture elements in the RGBcolorimetric system is (R, G, B), the following relationships existbetween chromaticity coordinates x, y in the X, Y, Z colorimetricsystem:x=(1.1302+1.6387r+O.6215g)/(6.7846−3.0157r−0.38579)  (58)y=(0.0601+0.9399r+4.5306g)/(6.7846−3.0157r−0.3857g)  (59).

Herein, as chromaticity represents an absolute proportion of a colorstimulation value without it being affected by lightness, it may be saidthat the nature of an object can be determined from the chromaticity ofits picture elements. Since0.35<x<O.40  (60)O.33<y<0.36  (61)in the case of skin color, it may be considered that if a pictureelement is within this range when chromaticity is determined for eachpicture element, that picture elements shows a person's skin, and thenumber of skin color picture elements in the block is increased by one.

After the number of skin color picture elements is obtained in this way,in the next step SB320, weighting coefficients are determined as in thecase of edge picture elements described above, and the luminancedistribution DY is reevaluated. Specifically, if the number of skincolor picture elements in each block is written as CNi (i=1-15), thetotal number SC of such elements is:

$\begin{matrix}{{SC} = {\sum\limits_{{i = 1},15}\;{CNi}}} & (62)\end{matrix}$Therefore the weighting coefficient KCi is expressed byKCi=CNi/SC  (63),and the luminance distribution DY reevaluated with this weighting may becalculated by

$\begin{matrix}{{DY} = {\sum\limits_{i = {1 \sim 15}}\;{{KCi}*{{dYi}.}}}} & (64)\end{matrix}$Also in this example, the luminance of picture elements determined to beskin color picture elements is not sampled, the total number of skincolor picture elements merely being used for determining block weightingcoefficients. Therefore an average feature amount that is not uneven canbe obtained for the block. In this case too, after the luminancedistribution has been reevaluated in this way, the contrast may becorrected and lightness compensated by the processing of the aforesaidsteps SB130-SB160. In the case of the photograph shown in FIG. 39, agirl appears in the center, and picture elements of the face, arms andlegs are determined to be skin color picture elements. Of course, thechromaticity may also be found and picture elements summed for othercolors.

Until now, the weighting coefficient was determined by one factor, butthe importance of each factor may also be added and the above processingapplied repeatedly. When the weighting coefficient of each block Bi(i=1-15) is Tji for a factor j (1=position in image, 2=edginess, 3=skincolor picture element number), the weighting Tji distributed amongblocks for each factor is a temporary weighting,

$\begin{matrix}{{{Sj} = {\sum\limits_{i = {1 \sim 15}}\;{Tji}}}{and}} & (65) \\{{{Kji} = {{Tji}/{Sj}}},} & (66)\end{matrix}$then the real weighting coefficient Ki in the block. Bi is given by

$\begin{matrix}{{Ki} = {\sum\limits_{j = {1 \sim 3}}\;{{Aj}*{{Kji}.}}}} & (67)\end{matrix}$Aj is a coefficient to represent the importance of each factor, and issuitably determined so that the total number is 1. If skin color isemphasized as an example, the settings A1=O.2, A2=O.2, A3=O.6 arepossible.

Next, the effect of this embodiment having the aforesaid constructionwill be described. First, a description will be given along the lines ofthe previous embodiment.

A photographic image is read by the scanner 11, and printed by theprinter 31. Specifically, when the operating system 21 a is running onthe computer 21, the image processing application 21 d is started, andreading of the photograph is started by the scanner 11. After the readimage data has been assimilated by the image processing application 21d, the luminance of picture elements is summed while thinning in thestep SB110. The summed luminance distribution dYi is reevaluated in thestep SB120 based on determined weightings corresponding to the positionof each block shown in FIG. 33 and FIG. 34, and ymax, ymin and ymed arecalculated in the step SB130 based on the reevaluated distribution DY.

In the next step SB140, the slope a and offset b which are emphasisparameters are computed based on equations (8) or (9), the γ value ofthe γ correction required for lightness compensation is calculated basedon equations (10)-(13), and the conversion data shown in FIG. 16 isgenerated in the step SB150. Finally, in the step SB160, the image datafor all picture elements is converted by referring to this conversiontable.

Using the weightings shown in FIG. 33, the weighting is higher forblocks near the center, so the summed luminance distribution is alsoweighted more heavily the closer it is to the center. Assume for examplethat a person is photographed at night using flash. Even if a goodoverall luminance distribution is obtained for the person as shown inFIG. 40( a) due to the effect of the flash, the part surrounding theperson is dark, and here the luminance distribution is shifted towardsthe dark side as shown in (b). In this case if an average were merelytaken, a luminance distribution shifted towards the dark side overallwould still be obtained as shown in (c), and if the contrast werecorrected and the lightness compensated, the image would only be madetoo light overall and would not be satisfactory.

However, if more weighting is given to the center block as shown in FIG.33, a luminance distribution DY is obtained which is strongly affectedby the luminance distribution in the center of the image as shown inFIG. 40( d). Hence, the intensity of image processing based on thisdistribution is no longer a matter of over-emphasizing contrast andover-compensating lightness.

Conversely, if a person is photographed with a backlight, the face willbe dark, and the background will be light, so a good luminancedistribution will not necessarily be obtained overall. Yet even in thiscase, by giving more weight to the luminance distribution of centerblocks where the face is dark as shown in FIG. 33, the dark luminancedistribution is reflected, and image processing is performed to increasecontrast and compensate lightness.

Due to the aforesaid processing, photographic image data read via thescanner 11 is processed automatically with optimum intensity, displayedon the display 32, and printed by the printer 31.

The computer 21 which is the core of image processing sums the luminancedistribution which a feature amount for each area while uniformlyselecting picture elements in the step SB110. In the step SB120, areevaluation is performed with weightings determined for each area, anda luminance distribution strongly influenced by the intrinsic luminancedistribution of the object can thus be obtained while uniform samplingis performed. The intensity of image processing is determined based onthis luminance distribution in the steps SB130-SB150, and the image datais converted in the step SB160. Hence, image processing is performedwith optimum intensity while the amount of processing is reduced.

Embodiment 3

Next, an embodiment will be described wherein the aforesaid imageprocessing indicator specifying unit comprises an evaluation unit thatobtains feature amounts based on a plurality of predetermined criteria,and the aforesaid processing unit which can convert image data by aplurality of methods, uses feature amounts according to the method.

FIG. 41 shows a block diagram of an image processing system whichapplies the image processing apparatus according to one embodiment ofthis invention. A typical hardware construction is similar to the systemshown in FIG. 2.

In FIG. 41, the image reader 10 outputs photographic image data as dotmatrix picture elements to the image processing apparatus 20C. Whenplural image processings are applied, the image processing apparatus 20Cdetermines plural feature amounts based on the optimum criterion foreach type of processing, and carries out each type of processing usingthe most appropriate feature amount. The image processing apparatus 20Coutputs the processed image data to the image output device 30, and theimage output device outputs the processed image in dot matrix pictureelements.

The image processing apparatus 20C first sums the image data accordingto plural evaluation criteria, and thereby obtains plural featureamounts. In this sense, the image processing apparatus 20C comprises anevaluation unit, and as it performs image processing based on featureamounts which have been selected according to the image processingcontent, it may be said to further comprise an image processing unit.

The acquisition of image feature amount of image and associated imageprocessing are performed by the computer 21 with an image processingprogram corresponding to the flowchart shown in FIG. 42. The flowchartshown in the figure corresponds to the preceding stage in the imageprocessing program, and processing is performed to sum image dataaccording to plural evaluation criteria.

According to this embodiment, the case will be described where twoevaluation criteria are used. Common points are that in both cases, notall picture elements are considered, picture elements are sampledaccording to predetermined criteria, and luminance is summed for thesampled picture elements.

What is different is that in one method, picture elements are sampleduniformly, whereas in the other method, edge picture elements areselected for sampling. Luminance summation results are describedhereafter, but here it should be noted that plural feature amounts areobtained according to different evaluation criteria by changing thesampling method.Uniform sampling means that luminance is summed for all picture elementsin the image, and the luminance distribution is determined for the imageas a whole. A feature amount is thereby obtained which is useful as areference when a scenic photograph is dark overall or contrast isnarrow. In the other method, as edge picture elements are a sharp partof the image, the luminance is summed for picture elements related tothe object in the image. Therefore provided that the object issufficiently light even if the background is dark, a feature amount isobtained which is useful as a reference when the image is sufficientlylight. According to this embodiment, these feature amounts are selectedautomatically according to the image processing method.

Referring to the flowchart of FIG. 42, in this summation processing,object picture elements are main scanned in the horizontal direction andauxiliary scanned in the vertical direction for image data comprisingdot matrix picture elements, and displaced, as shown in FIG. 24, and thesummation is performed by determining whether or not each pictureelement scanned is to be included in the sampling.

First, in a step SC110, the edginess of each picture element isdetermined. Specifically, the determination of edginess may be the sameas that of the aforesaid step SA110.

In the step SC120, the edginess is compared with the same thresholdvalue, and it is determined whether or not the variation is large. If asa result of comparison it is determined that edginess is large, it isdetermined that this picture element is an edge picture element, and theimage data for the picture element is sampled in a step SC130 and storedin the work area. The work area may be a RAM in the computer 21, or itmay be a hard disk 22.

In this embodiment, the object is sampled based on edginess, but ofcourse the method of sampling the object is not limited to this. Forexample, the chromaticity of each picture element may be found, andpicture elements for which the chromaticity is within a predeterminedrange can be sampled as the object.

Specifically, the processing of the step SC110 and step SC120 issubstituted by the processing of a step SC115 and a step SC125respectively, the chromaticity for each picture element is calculated inthe step SC115, and in the step SC125, it is determined whether or notthe x-y chromaticity that was converted based on RGB gradation data foreach picture element is within the range of skin color. If it is skincolor, the image data of the picture element is sampled in the stepSC130 and also stored similarly to work area.

On the other hand, in parallel with the aforesaid determination ofedginess, it is determined in a step SC140 whether or not this pictureelement is to be sampled by uniform sampling. Uniform sampling isidentical to the above. In the step SC140 it is determined whether ornot the picture element is to be sampled, and if so, the picture elementis sampled in a step SC150. It will be understood that the sampling ofimage data in the steps SC130, SC150 means the summation of luminancebased on this image data.

To perform this processing for all picture elements of the image data, apicture element to be processed is displaced in a step SC160, and theprocessing is repeated until it is determined in a step SC170 thatprocessing of all picture elements has finished.

Thereafter, a feature amount is obtained by using the summation resultDist_ave obtained by uniform sampling and the summation result Dist_edgobtained by edge picture element sampling according to the intendedimage processing method, and so optimum image processing based on thisfeature amount may be performed. As an example, FIG. 43 shows aflowchart for performing expansion of contrast and lightnesscompensation. The basic method of increasing contrast according to thisembodiment is as described hereabove. The correction parameters aregenerated in steps SC310-SC330. Then, it is determined in a step SC340whether contrast correction and lightness compensation are necessary,and if it is determined that they are, conversion of image data isperformed in a step SC350. In this step SC350, reference is made to aconversion table corresponding to equations (14)-(16) for image data(R0, G0, B0) for all picture elements, and the process of obtainingimage data (R, G, B) after conversion is repeated.

However in this case the luminance summation results are used asappropriate, and a feature amount is obtained comprising both ends andthe median of the luminance distribution. Contrast correction andlightness compensation are applied using this feature amount, but thisis not the only specific example of image processing, and there are alsovarious feature amounts which may be used. For example, saturationemphasis image processing may be performed as shown in FIG. 20.

In this case, the evaluation unit comprises a first stage of a programfor summing image data by the uniform sampling method (step SA410) up tothe acquisition of a feature amount which is a saturation emphasisindicator S (step SA420), and hardware for implementing this first stageof the program. The image processing unit comprises the latter stage ofthe program for performing conversion of image data (step SA440), andhardware for implementing this latter stage of the program.

Regarding contrast correction, lightness compensation and saturationemphasis, it is determined whether to perform image processing in eachcase. However, it is not absolutely necessary to make a choice as towhether or not to perform image processing. Specifically, a degree ofemphasis degree is set for each, and image processing may be performedwith the set degree of emphasis.

Next, the operation of this embodiment having the aforesaid constructionwill be described.

A photographic image is read by the scanner 11, and printed by theprinter 31. Specifically, when the operating system 21 a is running onthe computer 21, the image processing application 21 d is started, andreading of the photograph is started by the scanner 11. After the readimage data has been assimilated by the image processing application 21 dvia the operating system 21 a, the picture elements to be processed areset in initial positions. Next, the edginess is determined based onequations (1)-(3) in the step SC110, and the edginess is compared with athreshold value in the step SC120. When the edginess is large, it isdetermined that the picture element to be processed is an edge pictureelement, and image data for the corresponding picture element is storedin the work area in the step SC130. In the step SC140, it is determinedwhether or not picture elements to be processed are to sampleduniformly, and if so, in the step SC150, image data for these pictureelements is stored in the work area. The above processing is repeateduntil it is determined to have been performed for all picture elementsin the step SC170 while picture elements to be processed are displacedin the step SC160.

According to this embodiment, image data was stored in the work area,but from the viewpoints of memory capacity and processing time, it isnot absolutely necessary to store image data itself in the work area.Specifically, histograms of luminance distribution or saturationsubstitute value distribution for picture elements determined to bethose of the object are generated, and histogram information maytherefore be pre-stored in the steps SC120, SC150.

When summation has been performed on all picture elements, luminancedistribution histograms are found for the summation result Dist_aveobtained by uniform sampling and the summation result Dist_edg obtainedby edge picture element sampling in the step SC310. A parameter forexpansion processing is determined based on equations (8), (9) in thestep SC320, and a parameter for luminance compensation is determinedbased on equations (10)-(13) in the step SC330. These parameters arecompared with a predetermined threshold value in the step SC340, and ifit is determined that image processing should be performed, theluminance is converted based on the aforesaid parameter in the stepSC350. In this case, the luminance conversion table shown in FIG. 16 maybe generated to reduce the computing amount, and image data may beconverted based on equations (14)-(16).

Subsequently, the processed image data may be displayed on the display32 via the display driver 21 c, and if it is satisfactory, it may beprinted by the printer 31 via the printer driver 21 b. Specifically, theprinter driver 21 b inputs RGB gradation data with emphasized edges,performs rasterization corresponding to the print head area of theprinter 31 after a predetermined mapping, color converts rasterized datafrom RGB to CMYK, converts CMYK gradation data to binary data, andoutputs it to the printer 31.

Due to the above processing, photographic image data read via thescanner 11 is automatically subjected to the optimum contrast correctionand lightness compensation, displayed by the display 32, and printed bythe printer 31. Specifically, plural feature amounts are obtained usingplural evaluation criteria, and optimum image processing is performedwith different feature amounts according to the image processing methodsof contrast correction or lightness compensation.

However, the invention is not limited to this contrast correction andluminance compensation. In the case also of saturation emphasis, afeature amount is acquired by sampling saturation according to asuitable criterion depending on the saturation emphasis processing, andimage processing is performed based on this feature amount. In this way,optimum image processing is performed.

Hence, the computer 21 which is the core of image processing calculatesthe luminance distribution based on the summation results sampledaccording to different evaluation criteria in the step SC310, obtainsdifferent feature amounts from different luminance distributionhistograms in the steps SC320 and SC330, and converts image data basedon these feature amounts in the step SC350. This permits optimum imageprocessing.

Embodiment 4

Next, an embodiment will be described concerned mainly with imageevaluation which is an indicator for image processing.

FIG. 44 is a block diagram of an image processing system which performsimage processing by implementing an image evaluation method according toone embodiment of this invention. A typical hardware construction issimilar to the system shown in FIG. 2.

In FIG. 44, the image reader 10 outputs photographic image data as dotmatrix picture elements to an image processing apparatus 20D. The imageprocessing apparatus 20D calculates an evaluation result by summing theimage data after predetermined processing, determines the content andextent of image processing based on the evaluation result, and thenperforms image processing. The image processing apparatus 20D outputsthe processed image data to an image output device 30, and the imageoutput device outputs the processed image as dot matrix pictureelements.

The image processing apparatus 20D sums the image data beforehand, andcalculates an evaluation result for the corresponding image. The imagedata are summed individually using plural evaluation criteria, and arecombined by varying the weighting according to predetermined conditions.Therefore, the image processing apparatus 20D is an image dataevaluation unit.

Image evaluation and associated image processing are performed in thecomputer 21 by an image processing program corresponding to a flowchartsuch as is shown in FIG. 45. The flowchart shown in the figurecorresponds to the first stage of image evaluation in the imageprocessing program, and processing is performed to obtain apredetermined evaluation result by summing the image data according toplural evaluation criteria. The basic method of image evaluation issubstantially identical to that described above, however in thefollowing description, a simple evaluation result is obtained by usingimage evaluation options.

According to this embodiment, two of the evaluation criteria used willbe described. Common points are that in both cases, picture elements arethinned according to a predetermined criterion instead of consideringthe whole image, and the luminance of the sampled picture elements issummed.

A difference is that whereas in one case, picture elements are sampleduniformly, in the other case, edge picture elements are selected. Theluminance summation results are described hereafter, but the imageevaluation can be changed by changing the sampling method in this way.Uniform sampling of picture elements means that the luminance is summedfor all picture elements in the image, and the luminance distribution isdetermined for the image as a whole. The evaluation is therefore usefulas a reference when a scenic photograph is dark overall or contrast isnarrow. In the other method, as edge picture elements are a sharp partof the image, the luminance is summed for picture elements related tothe object in the image. For example, provided that the object issufficiently light even if the background is dark, an evaluation resultis obtained where the image is sufficiently light. According to thisembodiment, the image is determined by suitably combining two evaluationcriteria, i.e. a criterion selected by the operator or automaticprocessing.

Referring now to the flowchart of FIG. 45, in steps SD110-SD170, imagedata comprising dot matrix picture elements is main scanned in thehorizontal direction and auxiliary scanned in the vertical direction asshown in FIG. 24 as described in the aforesaid embodiment, and imagedata for edge picture elements and uniformly sampled picture elements isstored in the work area.

After luminance is summed for all picture elements concerned by thesevarious sampling methods, image evaluation options are input in a stepSD180. FIG. 46 shows an image evaluation option input screen displayedon the display 32. Three choices are available: portrait, scenicphotograph and automatic setting.

As shown in FIG. 47, weightings must be adjusted in order to generate ahistogram to evaluate the image which is obtained by combining aluminance histogram obtained by uniform sampling and a luminancehistogram obtained by edge sampling.

When the weighting coefficient k is adopted, a summation result Dist_Sumfor evaluation is obtained from the uniform sampling summation resultDist_ave and the edge sampling summation result Dist_edg by therelation:Dist_Sum=k×Dist_edg+(1−k)×Dist_ave  (68).The closer the weighting coefficient k approaches “0” the more the wholeimage is emphasized, and the closer it approaches “1”, the more theobject in the photograph is emphasized.As a result, after setting the options on the image evaluation optioninput screen shown in FIG. 46, the routine branches depending on theoption in a step SD190. When a portrait is selected, k is set to 0.8 ina step SD192, and when a scenic photograph is selected, k is set to 0.2in a step SD194.

The remaining option is the “AUTO” setting.

In this auto setting, based on the edge picture elements sampled asdescribed above, the image may be considered as a scenic photograph andthe weighting coefficient approaches “0” when there are few edge pictureelements, while the image may be considered as a portrait and theweighting coefficient approaches “1” when there are many edge pictureelementsBy using the sampling number x_edg of edge picture elements and theuniform sampling number x_ave, the weighting coefficient may be computedfrom:k−x_edg/(x_edg+x_ave)  (69)in a step SD196, and the summation result Dist_Sum used for evaluationcan be obtained.

After the summation result Dist_Sum used for evaluation is obtained,image evaluation can be performed. Of course, a further determinationmay be made using this summation result, and it may basically be variedas necessary depending on the image processing which uses the summationresult.

Subsequently, the optimum image processing is determined based on thesummation results and performed. As an example, FIG. 48 shows aflowchart to perform expansion of contrast and luminance compensation.In a basic technique to increase contrast according to this embodiment,the luminance distribution is found based on image data as in theaforesaid embodiment, and if this luminance distribution uses only partof the actual gradation width (255 gradations), it is expanded.

Therefore, in a step SD310, a histogram of luminance distribution isgenerated as the summation result Dist_Sum from the weightingcoefficient k, and the expansion width is determined in a step SD320.The expansion width determining method is as described above. Thegeneration of a predetermined conversion chart corresponds to theexpansion width determination processing of the step S320, so image datacan be modified by referring to this chart. By expanding the luminancerange, not only is the contrast emphasized, but it is also very usefulto adjust the luminance range at the same time. Therefore the luminanceof the image is determined in a step SD330, and a parameter is generatedfor the correction.

In this case, the summation results used for determining the image areused as evaluation criteria. Contrast correction and luminancecompensation are performed, but specific examples of image processingare not limited to this, and therefore various summation results may beused as evaluation criteria.

Examples of image processing are saturation emphasis or edge emphasis.In these cases, image vividness and image sharpness need to beevaluated. The summation may be performed by the above methods.

It may be determined whether image processing is performed regardingcontrast correction, luminance compensation, saturation emphasis andedge emphasis, but it is not absolutely necessary to make a choice as towhether or not to perform image processing. Specifically, a degree ofemphasis is set for each, and image processing is performed using thisset degree of emphasis.

Next, the action of this embodiment having the aforesaid constructionwill be described.

Assume that a photographic image is read with the scanner 11, andprinted by the printer 31. When the operating system 21 a is running,the image processing application 21 d is started by the computer 21, andreading of a photograph is started relative to scanner 11. After theread image data has been assimilated via the operating system 21 a by animage processing application 21 d, the picture elements to be processedare set in initial positions. Next, the edginess is determined based onequations (1)-(3) in the step SD110, and the edginess is compared with athreshold value in the step SD120. When the edginess is large, it isdetermined that the picture element to be processed is an edge pictureelement, and image data for the corresponding picture element is storedin the work area in the step SD130. In the step SD140, it is determinedwhether or not picture elements to be processed are to sampleduniformly, and if so, in the step SD150, image data for these pictureelements is stored in the work area. The above processing is repeateduntil it is determined to have been performed for all picture elementsin the step SD170 while picture elements to be processed are displacedin the step SD160.

When this has been performed for all picture elements, image datasampled according to different evaluation criteria is stored indifferent work areas, and in a step SD180, image evaluation options areinput. The operator, looking at the image, may select either a portraitor scenic photograph is that can be determined. If it cannot bedetermined or when it is desired to fully automate operations, theoperator selects the auto setting. When a portrait is selected, theweighting coefficient k becomes “O.8”, which puts more weight on thesummation results for the edge picture elements. When a scenicphotograph is selected, the weighting coefficient k becomes “O.2”, whichputs more weight on uniformly sampled summation results. When the autosetting is selected, the weighting coefficient k is set according to theproportion of edge picture elements. However, in any of these casesplural evaluation criteria are adopted using the weighting coefficientk. This permits a flexible evaluation not limited to just one criterion.

According to this embodiment, image data was stored in the work area,but from the viewpoints of memory capacity and processing time, it isnot absolutely necessary to store image data itself in the work area.Specifically, histograms of luminance distribution or saturationsubstitute value distribution for sampled picture elements aregenerated, and histogram information may therefore be prestored in thesteps SD120, SD150.

When contrast correction and luminance compensation are performedautomatically, a weighting coefficient is used, a histogram of luminancedistribution is found in the steps SD120, SD150, SD310, a parameter forexpansion processing is determined based on equations (8), (9) in thestep SD320, and a parameter for luminance compensation is determinedbased on equations (10)-(13) in the step SD330. These parameters arecompared with a predetermined threshold value in a step SD340, and if itis determined that image processing should be performed, the luminanceis converted based on the aforesaid parameter in a step SD350. In thiscase, a luminance conversion table shown in FIG. 16 may be generated toreduce the computing amount, and image data may be converted based onequations (14)-(16).

Subsequently, the processed image data may be displayed on the display32 via a display driver 21 c, and if it is satisfactory, it is printedby the printer 31 via a printer driver 21 b. In other words the printerdriver 21 b inputs RGB gradation data with emphasized edges, performsrasterization corresponding to the print head area of the printer 31after a predetermined mapping, saturation converts rasterized data fromRGB to CMYK, converts CMYK gradation data to binary data, and outputs itto the printer 31.

The photographic image data read via the scanner 11 and printed by theprinter 31 or displayed by the display 32, is therefore automaticallysubjected to optimum contrast correction and luminance compensation.More specifically, the image can be determined more flexibly by adoptingplural evaluation criteria, and optimum image processing realized bycontrast correction and luminance compensation.

However, the invention is not limited to this contrast correction andluminance compensation. Saturation and edginess are also sampled byplural evaluation criteria and summed, and weighting coefficients areadjusted and added. Therefore, image processing is performed after aflexible determination which is not limited to a single evaluationcriterion.

In this way, the computer 21 which is the core of image processing firstsamples image data for picture elements according to differentevaluation criteria in the steps SD120, SD140, and determines theweighting coefficient k in the steps SD192-SD196 based on imageevaluation options input in the step SD180. It then generates aluminance distribution histogram by adding summation results in the stepSD310 using these weighting coefficients, and performs optimum imageprocessing in the steps SD310-SD350.

What is claimed is:
 1. A method for image processing image data of animage comprising a plurality of picture elements, the method comprising:while scanning the image data by displacing a target picture element inthe image, determining the target picture element to be sampled or notaccording to one or a plurality of different sampling methods each ofwhich has its own independent criterion for sampling or not, samplingthe target picture element which is determined to be sampled accordingto each of the sampling methods independently for each of the samplingmethods thus obtaining sampled results as thinned image dataindependently for each of the sampling methods, after scanning the imagedata, obtaining feature amounts of the thinned image data using thesampled results for each of the sampling methods, and acquiringparameters for the image processing using the feature amounts.
 2. Amethod for image processing image data according to claim 1, wherein aninformation to prepare histogram of the target picture element iscalculated and stored instead of storing the image data.
 3. A method forimage processing image data according to claim 1, wherein while scanningthe image data, the plurality of different sampling methods is performedin parallel.
 4. A method for image processing image data according toclaim 1, wherein while scanning the image data, the target pictureelement is set in an initial position, and the target picture element isdisplaced to a next picture element until predetermined number of thepicture elements have been finished.
 5. A method for image processingimage data according to claim 1, wherein the target picture element isdetermined based on an edginess of the picture elements.
 6. An imageprocessing device comprising: a first unit that inputs the image data;and a second unit that processes the image data using the featureamounts obtained according to claim
 1. 7. A method for image processingimage data, the method comprising: scanning the image data, obtaining afirst sampled data set thinned by using a first sampling method and asecond sampled data set thinned by using a second sampling method thatis different from the first sampling method while scanning the imagedata once, each of the first sampling method and the second samplingmethod having its own independent criterion for sampling or not, andafter the scanning the image data, acquiring a first feature amount ofthe scanned image data from the first sampled data set and a secondfeature amount of the scanned image from the second sampled data set,wherein each of target picture elements of the image data is determinedto be sampled or not according to at least one of the first samplingmethod and the second sampling method, and each of the target pictureelements, which is determined to be sampled, is sampled independentlyfor each of the first sampling method and the second sampling method,thus obtaining the first sampled data set and the second sampled dataset as thinned image data sets independently for each of the firstsampling method and the second sampling method.
 8. A method for imageprocessing image data according to claim 7, wherein the image datacomprises picture elements and each of the first sampled data set andthe second sampled data set comprises an information to preparehistogram of scanned picture elements, and the method further comprises:while scanning the image data, calculating and storing the histograminformation instead of storing the image data.
 9. A method for imageprocessing image data according to claim 7, wherein the image datacomprises picture elements and the method further comprises: determiningwhether to sample the picture elements based on an edginess of thepicture elements.
 10. A method for image processing image data accordingto claim 7, wherein while scanning the image data, the first and secondsampling methods are performed in parallel.
 11. A method for imageprocessing image data according to claim 7, wherein while scanning theimage data, a target picture element is set in an initial position, thetarget picture element is determined to be sampled according to each ofthe first and second sampling methods, and the target picture element isdisplaced to a next picture element until a predetermined number ofpicture elements has been finished.
 12. A method for image processingimage data according to claim 7, wherein the first feature amount andthe second feature amount are used for processing the image data.
 13. Animage processing device comprising: a first unit that inputs the imagedata; and a second unit that processes the image data using the firstsampled data set and the second sampled data set obtained according toclaim 7.